[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:27 dist-test-slave-0f28 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:27 dist-test-slave-0f28 krb5kdc[399](info): set up 2 sockets
Jun 19 16:35:27 dist-test-slave-0f28 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 16:35:28.157692 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:28.276690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:28.384235 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:28.384473 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:28.384511 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:28.388460 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:28.388511 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:28.388530 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:28.388547 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:28.388610 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:28.393608 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:28.394541 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:28.395535 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:28.404640 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:28.404639 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:28.404726 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:28.404747 412 server_base.cc:1034] running on GCE node
I20260619 16:35:28.405540 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:28.406615 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:28.407791 412 hybrid_clock.cc:648] HybridClock initialized: now 1781886928407735 us; error 77 us; skew 500 ppm
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:28.410446 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:28.411244 412 webserver.cc:460] Webserver started at http://127.0.98.62:35031/ using document root <none> and password file <none>
I20260619 16:35:28.411773 412 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:28.411860 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:28.412099 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:28.414119 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "68da4918a38f474e9a31cf7790d07480"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "30ec2557461f26c2b236d9619de25a0b"
server_key_iv: "cdaca5e6b4c774f03509edb868746e09"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.414659 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "68da4918a38f474e9a31cf7790d07480"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "30ec2557461f26c2b236d9619de25a0b"
server_key_iv: "cdaca5e6b4c774f03509edb868746e09"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.418222 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:28.420415 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.421631 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:28.421762 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "68da4918a38f474e9a31cf7790d07480"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "30ec2557461f26c2b236d9619de25a0b"
server_key_iv: "cdaca5e6b4c774f03509edb868746e09"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.421873 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:28.432284 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:28.434957 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:28.435149 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:28.441780 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33175
I20260619 16:35:28.441809 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33175 every 8 connection(s)
I20260619 16:35:28.442656 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:28.445209 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:28.450384 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480: Bootstrap starting.
I20260619 16:35:28.452435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 412
I20260619 16:35:28.452598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:28.452858 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:28.453658 480 log.cc:824] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:28.454033 392 external_mini_cluster.cc:1428] Setting key 1ac60f7d6c350ce8981cf34bb7c87021
I20260619 16:35:28.455488 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480: No bootstrap required, opened a new log
I20260619 16:35:28.459928 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68da4918a38f474e9a31cf7790d07480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33175 } }
I20260619 16:35:28.460157 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:28.460245 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68da4918a38f474e9a31cf7790d07480, State: Initialized, Role: FOLLOWER
I20260619 16:35:28.460654 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68da4918a38f474e9a31cf7790d07480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33175 } }
I20260619 16:35:28.460806 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:28.460891 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:28.461006 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:28.461946 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68da4918a38f474e9a31cf7790d07480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33175 } }
I20260619 16:35:28.462284 480 leader_election.cc:302] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68da4918a38f474e9a31cf7790d07480; no voters:
I20260619 16:35:28.462631 480 leader_election.cc:288] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:28.462679 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:28.462846 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [term 1 LEADER]: Becoming Leader. State: Replica: 68da4918a38f474e9a31cf7790d07480, State: Running, Role: LEADER
I20260619 16:35:28.463178 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68da4918a38f474e9a31cf7790d07480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33175 } }
I20260619 16:35:28.463840 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:28.464303 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68da4918a38f474e9a31cf7790d07480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68da4918a38f474e9a31cf7790d07480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33175 } } }
I20260619 16:35:28.464481 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:28.464700 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68da4918a38f474e9a31cf7790d07480. Latest consensus state: current_term: 1 leader_uuid: "68da4918a38f474e9a31cf7790d07480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68da4918a38f474e9a31cf7790d07480" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33175 } } }
I20260619 16:35:28.464804 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:28.465147 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:28.467975 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:28.474228 491 catalog_manager.cc:1347] Generated new cluster ID: 7a7dbd7d90154132b9ddfcc5bfecf8e6
I20260619 16:35:28.474292 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:28.480376 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:28.456841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.0.98.62:33175)
0619 16:35:28.457247 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:28.457262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:28.458507 (+ 1245us) server_negotiation.cc:407] Connection header received
0619 16:35:28.461443 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:28.461455 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:28.461729 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:28.462036 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:28.464974 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.466519 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.468239 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.468925 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.469026 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:28.474088 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:28.474095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:28.474135 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:28.474169 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:28.476428 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:28.477064 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:28.477071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:28.477076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:28.477233 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:28.477658 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:28.477662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:28.477664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:28.478181 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:28.478352 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:28.479156 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 16:35:28.479458 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260619 16:35:28.482862 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:28.483945 491 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:28.490285 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68da4918a38f474e9a31cf7790d07480: Generated new TSK 0
I20260619 16:35:28.490940 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:28.520524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33175
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:28.630911 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:28.631198 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:28.631275 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:28.635453 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:28.635552 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:28.635645 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:28.641561 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:28.642572 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:28.643554 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:28.652987 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:28.653102 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:28.653149 506 server_base.cc:1034] running on GCE node
W20260619 16:35:28.653102 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:28.653679 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:28.654536 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:28.655761 506 hybrid_clock.cc:648] HybridClock initialized: now 1781886928655751 us; error 28 us; skew 500 ppm
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:28.658533 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:28.659356 506 webserver.cc:460] Webserver started at http://127.0.98.1:42603/ using document root <none> and password file <none>
I20260619 16:35:28.659893 506 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:28.659983 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:28.660207 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:28.662184 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "baea433f6ea04a2a9821bdf613989633"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "5e7f652db86894013c0eb52aa6ca7c1c"
server_key_iv: "8740a98ce4e89ffbd6d048b80301cf9f"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.662745 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "baea433f6ea04a2a9821bdf613989633"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "5e7f652db86894013c0eb52aa6ca7c1c"
server_key_iv: "8740a98ce4e89ffbd6d048b80301cf9f"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.666570 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:35:28.668892 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.670176 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:28.670315 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "baea433f6ea04a2a9821bdf613989633"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "5e7f652db86894013c0eb52aa6ca7c1c"
server_key_iv: "8740a98ce4e89ffbd6d048b80301cf9f"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.670421 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:28.685845 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:28.688570 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:28.688753 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:28.689275 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:28.690028 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:28.690099 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.690167 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:28.690210 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.698508 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34439
I20260619 16:35:28.698534 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34439 every 8 connection(s)
I20260619 16:35:28.699345 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:28.706746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 506
I20260619 16:35:28.706867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:28.707162 392 external_mini_cluster.cc:1428] Setting key 74554f079242be2b16249f008ce05636
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:28.713500 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:28.701101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38265 (local address 127.0.98.62:33175)
0619 16:35:28.701239 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:28.701242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:28.702029 (+ 787us) server_negotiation.cc:407] Connection header received
0619 16:35:28.703247 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:28.703249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:28.703294 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:28.703366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:28.704455 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.704981 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.706202 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.706446 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.706560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:28.709178 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:28.709184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:28.709190 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:28.709221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:28.711182 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:28.711803 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:28.711805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:28.711806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:28.711877 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:28.712287 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:28.712290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:28.712291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:28.712463 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:28.712549 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:28.713202 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 16:35:28.713373 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:35:28.714624 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:28.701494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33175 (local address 127.0.98.1:38265)
0619 16:35:28.701887 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:28.701929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:28.703055 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:28.703522 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:28.703528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:28.703858 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:28.704303 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:28.704316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.705120 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.705130 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:28.706088 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:28.706098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.706596 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.706599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:28.706732 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:28.707329 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:28.707345 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:28.709014 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:28.711299 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:28.711305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:28.711321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:28.711701 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:28.711970 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:28.711976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:28.711981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:28.712158 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:28.712576 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:28.712585 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:28.712932 (+ 347us) client_negotiation.cc:769] Sending connection context
0619 16:35:28.713143 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 16:35:28.713513 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260619 16:35:28.715880 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33175
I20260619 16:35:28.716120 635 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:28.716564 635 heartbeater.cc:511] Master 127.0.98.62:33175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:35:28.717962 444 ts_manager.cc:194] Registered new tserver with Master: baea433f6ea04a2a9821bdf613989633 (127.0.98.1:34439)
I20260619 16:35:28.719134 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:28.735942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33175
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:28.843384 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:28.843614 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:28.843662 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:28.847577 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:28.847630 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:28.847692 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:28.853020 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:28.854053 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:28.855018 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:28.864413 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:28.864614 640 server_base.cc:1034] running on GCE node
W20260619 16:35:28.864771 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:28.864766 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:28.865355 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:28.866149 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:28.867295 640 hybrid_clock.cc:648] HybridClock initialized: now 1781886928867269 us; error 48 us; skew 500 ppm
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:28.870612 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:28.871691 640 webserver.cc:460] Webserver started at http://127.0.98.2:42013/ using document root <none> and password file <none>
I20260619 16:35:28.872324 640 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:28.872386 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:28.872539 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:28.874589 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "b894d88815324edf80a156f81fecb9ab"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "9de64fd15b04caaf633307972031092d"
server_key_iv: "299f5f3b9a628b54a34a85e0f4104b57"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.875082 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b894d88815324edf80a156f81fecb9ab"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "9de64fd15b04caaf633307972031092d"
server_key_iv: "299f5f3b9a628b54a34a85e0f4104b57"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.879171 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:35:28.881712 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.882886 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:28.882978 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "b894d88815324edf80a156f81fecb9ab"
format_stamp: "Formatted at 2026-06-19 16:35:28 on dist-test-slave-0f28"
server_key: "9de64fd15b04caaf633307972031092d"
server_key_iv: "299f5f3b9a628b54a34a85e0f4104b57"
server_key_version: "encryptionkey@0"
I20260619 16:35:28.883116 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:28.899037 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:28.902334 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:28.902532 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:28.903080 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:28.903856 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:28.903901 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.903966 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:28.904000 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:28.913831 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43205
I20260619 16:35:28.913893 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43205 every 8 connection(s)
I20260619 16:35:28.914707 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:28.921525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 640
I20260619 16:35:28.921657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:28.921934 392 external_mini_cluster.cc:1428] Setting key b7cc65fb712ee08549192dbd0a1b2307
Jun 19 16:35:28 dist-test-slave-0f28 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:28.928731 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:28.916508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55353 (local address 127.0.98.62:33175)
0619 16:35:28.916635 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:28.916639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:28.917502 (+ 863us) server_negotiation.cc:407] Connection header received
0619 16:35:28.918833 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:28.918836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:28.918895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:28.918995 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:28.920127 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.920642 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.921960 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.922228 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.922339 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:28.924879 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:28.924885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:28.924892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:28.924915 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:28.926554 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:28.927214 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:28.927217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:28.927218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:28.927299 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:28.927690 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:28.927692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:28.927693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:28.927855 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:28.927982 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:28.928454 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 16:35:28.928569 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:35:28.929584 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:28.916890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33175 (local address 127.0.98.2:55353)
0619 16:35:28.917357 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:28.917409 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:28.918623 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:28.919168 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:28.919175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:28.919517 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:28.919982 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:28.919995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.920766 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.920771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:28.921840 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:28.921849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:28.922371 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:28.922375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:28.922599 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:28.923235 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:28.923256 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:28.924726 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:28.926675 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:28.926683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:28.926700 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:28.927061 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:28.927401 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:28.927407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:28.927412 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:28.927594 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:28.927978 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:28.927985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:28.928218 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 16:35:28.928401 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:35:28.928741 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20260619 16:35:28.930825 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33175
I20260619 16:35:28.931067 769 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:28.931519 769 heartbeater.cc:511] Master 127.0.98.62:33175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:28.932561 444 ts_manager.cc:194] Registered new tserver with Master: b894d88815324edf80a156f81fecb9ab (127.0.98.2:43205)
I20260619 16:35:28.933132 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:28.948764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33175
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:29.055544 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:29.055775 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:29.055815 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:29.059755 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:29.059804 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:29.059890 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:29.065127 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:29.066136 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:29.067018 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:29.076121 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:29.076112 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:29.076265 774 server_base.cc:1034] running on GCE node
W20260619 16:35:29.076112 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:29.076805 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:29.077389 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:29.078588 774 hybrid_clock.cc:648] HybridClock initialized: now 1781886929078558 us; error 43 us; skew 500 ppm
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:29.081557 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:29.082427 774 webserver.cc:460] Webserver started at http://127.0.98.3:34841/ using document root <none> and password file <none>
I20260619 16:35:29.082964 774 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:29.083024 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:29.083249 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:29.085167 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "eff2c0b5f1db4c92b01213c7484c0748"
format_stamp: "Formatted at 2026-06-19 16:35:29 on dist-test-slave-0f28"
server_key: "dcab3859fac9243047ff0731a4343e42"
server_key_iv: "e6761a64d1c00f51091b9203ab1ee906"
server_key_version: "encryptionkey@0"
I20260619 16:35:29.085772 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eff2c0b5f1db4c92b01213c7484c0748"
format_stamp: "Formatted at 2026-06-19 16:35:29 on dist-test-slave-0f28"
server_key: "dcab3859fac9243047ff0731a4343e42"
server_key_iv: "e6761a64d1c00f51091b9203ab1ee906"
server_key_version: "encryptionkey@0"
I20260619 16:35:29.089416 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:35:29.092156 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:29.093451 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:29.093585 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "eff2c0b5f1db4c92b01213c7484c0748"
format_stamp: "Formatted at 2026-06-19 16:35:29 on dist-test-slave-0f28"
server_key: "dcab3859fac9243047ff0731a4343e42"
server_key_iv: "e6761a64d1c00f51091b9203ab1ee906"
server_key_version: "encryptionkey@0"
I20260619 16:35:29.093695 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:29.113709 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:29.116393 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:29.116583 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:29.117105 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:29.117903 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:29.117976 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:29.118042 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:29.118093 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:29.128021 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33417
I20260619 16:35:29.128062 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33417 every 8 connection(s)
I20260619 16:35:29.128888 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:29.134284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 774
I20260619 16:35:29.134397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:29.134667 392 external_mini_cluster.cc:1428] Setting key f6811273d0e30e1a6dd52d1b8e1e1468
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:29.142745 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:29.130749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42653 (local address 127.0.98.62:33175)
0619 16:35:29.130891 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:29.130896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:29.131792 (+ 896us) server_negotiation.cc:407] Connection header received
0619 16:35:29.133166 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:29.133169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:29.133214 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:29.133291 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:29.134820 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.135452 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.137021 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.137228 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.137396 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:29.139691 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:29.139694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:29.139696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:29.139719 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:29.140991 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:29.141503 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:29.141506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:29.141507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:29.141579 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:29.141893 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:29.141895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:29.141896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:29.142053 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:29.142127 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:29.142542 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:35:29.142635 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:35:29.143621 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:29.131169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33175 (local address 127.0.98.3:42653)
0619 16:35:29.131635 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:29.131679 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:29.132964 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:29.133532 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:29.133539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:29.133921 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:29.134637 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:29.134658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.135588 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.135594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:29.136878 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:29.136890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.137438 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.137442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:29.137607 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:29.138145 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:29.138163 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:29.139544 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:29.141102 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:29.141106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:29.141116 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:29.141421 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:29.141691 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:29.141693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:29.141694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:29.141817 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:29.142139 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:29.142144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:29.142351 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 16:35:29.142495 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:35:29.142755 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":106,"threads_started":1}
I20260619 16:35:29.144621 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33175
I20260619 16:35:29.144836 903 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:29.145292 903 heartbeater.cc:511] Master 127.0.98.62:33175 requested a full tablet report, sending...
I20260619 16:35:29.146301 444 ts_manager.cc:194] Registered new tserver with Master: eff2c0b5f1db4c92b01213c7484c0748 (127.0.98.3:33417)
I20260619 16:35:29.146844 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42653
I20260619 16:35:29.148983 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:29.175221 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:29.164913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.0.98.1:34439)
0619 16:35:29.165260 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:29.165277 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:29.165320 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:35:29.165395 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:29.165400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:29.165647 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:29.165765 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:29.166520 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.167286 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.168175 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.168555 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.168657 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:29.171994 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:29.172004 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:29.172020 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:29.172056 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:29.173606 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:29.174077 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:29.174082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:29.174084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:29.174160 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:29.174374 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:29.174375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:29.174376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:29.174591 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:29.174741 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:29.174887 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 16:35:29.175076 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":145,"threads_started":1}
W20260619 16:35:29.176155 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52676
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:29.201726 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:29.193383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.0.98.1:34439)
0619 16:35:29.193517 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:29.193520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:29.193567 (+ 47us) server_negotiation.cc:407] Connection header received
0619 16:35:29.193701 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:29.193704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:29.193743 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:29.193820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:29.194453 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.195016 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.195867 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:29.196090 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:29.196174 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:29.198723 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:29.198727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:29.198729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:29.198750 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:29.200047 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:29.200457 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:29.200459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:29.200461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:29.200527 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:29.200893 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:29.200895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:29.200896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:29.201067 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:29.201163 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:29.201463 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 16:35:29.201551 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:35:29.202706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 506
I20260619 16:35:29.209704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 640
I20260619 16:35:29.217087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 774
I20260619 16:35:29.222407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 412
2026-06-19T16:35:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2136 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[917](info): set up 2 sockets
Jun 19 16:35:29 dist-test-slave-0f28 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 16:35:30.260987 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:30 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:30.379510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:30.485440 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:30.485752 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:30.485833 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:30.489838 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:30.489912 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:30.489949 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:30.490006 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:30.490046 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:30.495170 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:30.496202 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:30.497216 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:30.505666 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:30.505754 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:30.505752 930 server_base.cc:1034] running on GCE node
W20260619 16:35:30.505955 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:30.506476 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:30.507491 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:30.508656 930 hybrid_clock.cc:648] HybridClock initialized: now 1781886930508637 us; error 30 us; skew 500 ppm
Jun 19 16:35:30 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:30.511335 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:30.512084 930 webserver.cc:460] Webserver started at http://127.0.98.62:43491/ using document root <none> and password file <none>
I20260619 16:35:30.512578 930 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:30.512674 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:30.512861 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:30.515261 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "ea9e8e6781104f7184ddf457ab69125d"
format_stamp: "Formatted at 2026-06-19 16:35:30 on dist-test-slave-0f28"
server_key: "8d9426582c65317661f5fcf8d06edf5d"
server_key_iv: "35c5df91ff32c5d03f27448f64c052fe"
server_key_version: "encryptionkey@0"
I20260619 16:35:30.515758 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea9e8e6781104f7184ddf457ab69125d"
format_stamp: "Formatted at 2026-06-19 16:35:30 on dist-test-slave-0f28"
server_key: "8d9426582c65317661f5fcf8d06edf5d"
server_key_iv: "35c5df91ff32c5d03f27448f64c052fe"
server_key_version: "encryptionkey@0"
I20260619 16:35:30.519295 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:30.521570 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:30.522732 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:30.522850 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "ea9e8e6781104f7184ddf457ab69125d"
format_stamp: "Formatted at 2026-06-19 16:35:30 on dist-test-slave-0f28"
server_key: "8d9426582c65317661f5fcf8d06edf5d"
server_key_iv: "35c5df91ff32c5d03f27448f64c052fe"
server_key_version: "encryptionkey@0"
I20260619 16:35:30.522946 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:30.541817 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:30.544888 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:30.545084 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:30.551401 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36827
I20260619 16:35:30.551469 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36827 every 8 connection(s)
I20260619 16:35:30.552306 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:30.554908 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:30.555148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 930
I20260619 16:35:30.555258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:30.555495 392 external_mini_cluster.cc:1428] Setting key a7be0c72064f1b5c4bdfd6d2fa44f577
I20260619 16:35:30.561631 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d: Bootstrap starting.
I20260619 16:35:30.563949 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d: Neither blocks nor log segments found. Creating new log.
Jun 19 16:35:30 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:30.564690 998 log.cc:824] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:30.566603 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d: No bootstrap required, opened a new log
I20260619 16:35:30.569446 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:30.556782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.0.98.62:36827)
0619 16:35:30.557184 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:30.557202 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:30.557245 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:35:30.558042 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:30.558065 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:30.558468 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:30.558815 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:30.559452 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:30.560913 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:30.561908 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:30.562545 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:30.562708 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:30.565122 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:30.565127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:30.565139 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:30.565165 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:30.566802 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:30.567231 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:30.567236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:30.567244 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:30.567350 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:30.567654 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:30.567659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:30.567661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:30.568015 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:30.568238 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:30.568468 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 16:35:30.568735 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260619 16:35:30.570051 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e8e6781104f7184ddf457ab69125d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260619 16:35:30.570274 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:30.570327 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9e8e6781104f7184ddf457ab69125d, State: Initialized, Role: FOLLOWER
I20260619 16:35:30.570775 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e8e6781104f7184ddf457ab69125d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260619 16:35:30.570924 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:30.571004 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:30.571122 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:30.571988 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e8e6781104f7184ddf457ab69125d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260619 16:35:30.572297 998 leader_election.cc:302] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9e8e6781104f7184ddf457ab69125d; no voters:
I20260619 16:35:30.572566 998 leader_election.cc:288] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:30.572681 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:30.572912 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [term 1 LEADER]: Becoming Leader. State: Replica: ea9e8e6781104f7184ddf457ab69125d, State: Running, Role: LEADER
I20260619 16:35:30.573220 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e8e6781104f7184ddf457ab69125d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260619 16:35:30.573729 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:30.574896 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9e8e6781104f7184ddf457ab69125d. Latest consensus state: current_term: 1 leader_uuid: "ea9e8e6781104f7184ddf457ab69125d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e8e6781104f7184ddf457ab69125d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260619 16:35:30.575017 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:30.575409 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9e8e6781104f7184ddf457ab69125d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9e8e6781104f7184ddf457ab69125d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260619 16:35:30.576000 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d [sys.catalog]: This master's current role is: LEADER
W20260619 16:35:30.578354 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:35:30.578421 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:35:30.578477 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:30.579147 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:30.584353 1012 catalog_manager.cc:1347] Generated new cluster ID: bdd302e3f9c6442eaf0e3fb136406690
I20260619 16:35:30.584426 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:30.591792 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:30.592710 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:30.598169 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9e8e6781104f7184ddf457ab69125d: Generated new TSK 0
I20260619 16:35:30.598821 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:30.630565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:30.742316 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:30.742555 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:30.742599 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:30.746771 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:30.746824 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:30.746917 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:30.752022 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:30.753391 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:30.754565 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:30.764487 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:30.764472 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:30.764451 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:30.765040 1024 server_base.cc:1034] running on GCE node
I20260619 16:35:30.765493 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:30.766152 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:30.767360 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781886930767324 us; error 52 us; skew 500 ppm
Jun 19 16:35:30 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:30.770360 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:30.771229 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46691/ using document root <none> and password file <none>
I20260619 16:35:30.771790 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:30.771883 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:30.772106 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:30.774245 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "2709ef5060e14df2bdd594a4b710c6f2"
format_stamp: "Formatted at 2026-06-19 16:35:30 on dist-test-slave-0f28"
server_key: "287da81fb59ae53e860ef19d9c4b9106"
server_key_iv: "5b41e868b1c18d231c8f1a2550ff2ffa"
server_key_version: "encryptionkey@0"
I20260619 16:35:30.774856 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2709ef5060e14df2bdd594a4b710c6f2"
format_stamp: "Formatted at 2026-06-19 16:35:30 on dist-test-slave-0f28"
server_key: "287da81fb59ae53e860ef19d9c4b9106"
server_key_iv: "5b41e868b1c18d231c8f1a2550ff2ffa"
server_key_version: "encryptionkey@0"
I20260619 16:35:30.778683 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:35:30.781152 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:30.782395 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:30.782528 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "2709ef5060e14df2bdd594a4b710c6f2"
format_stamp: "Formatted at 2026-06-19 16:35:30 on dist-test-slave-0f28"
server_key: "287da81fb59ae53e860ef19d9c4b9106"
server_key_iv: "5b41e868b1c18d231c8f1a2550ff2ffa"
server_key_version: "encryptionkey@0"
I20260619 16:35:30.782639 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:30.799783 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:30.802763 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:30.802979 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:30.803547 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:30.804381 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:30.804446 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:30.804512 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:30.804563 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:30.813894 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40065
I20260619 16:35:30.813966 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40065 every 8 connection(s)
I20260619 16:35:30.814802 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:30.815936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1024
I20260619 16:35:30.816031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:30.816291 392 external_mini_cluster.cc:1428] Setting key 025782359fb0cf14ac24dbb7b661bb2c
Jun 19 16:35:30 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:35:30.829823 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:30.817447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48329 (local address 127.0.98.62:36827)
0619 16:35:30.817576 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:30.817583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:30.817932 (+ 349us) server_negotiation.cc:407] Connection header received
0619 16:35:30.819479 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:30.819482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:30.819524 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:30.819592 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:30.821161 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:30.821775 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:30.823578 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:30.823809 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:30.823888 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:30.826721 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:30.826725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:30.826728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:30.826753 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:30.828069 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:30.828512 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:30.828515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:30.828516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:30.828587 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:30.828868 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:30.828870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:30.828871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:30.829035 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:30.829133 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:30.829608 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 16:35:30.829702 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:35:30.830627 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:30.817165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:48329)
0619 16:35:30.817772 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:30.817829 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:30.819233 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:30.819799 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:30.819810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:30.820346 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:30.821005 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:30.821028 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:30.821893 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:30.821899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:30.823444 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:30.823459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:30.823968 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:30.823974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:30.824205 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:30.824855 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:30.824881 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:30.826555 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:30.828164 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:30.828168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:30.828179 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:30.828426 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:30.828664 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:30.828667 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:30.828668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:30.828788 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:30.829145 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:30.829149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:30.829421 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 16:35:30.829572 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 16:35:30.829804 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":158,"threads_started":1}
I20260619 16:35:30.831804 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260619 16:35:30.832029 1153 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:30.832485 1153 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260619 16:35:30.833940 962 ts_manager.cc:194] Registered new tserver with Master: 2709ef5060e14df2bdd594a4b710c6f2 (127.0.98.1:40065)
I20260619 16:35:30.835047 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:35:31.837150 1153 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260619 16:35:32.278708 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 1.461s user 0.003s sys 0.000s
I20260619 16:35:32.279371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:32.397236 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:32.397645 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:32.397727 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:32.401654 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:32.401734 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:32.401830 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:32.407145 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:32.408237 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:32.409861 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:35:32.419288 1158 server_base.cc:1034] running on GCE node
W20260619 16:35:32.419296 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:32.419270 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:32.419354 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:32.420012 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:32.420665 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:32.421830 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781886932421801 us; error 43 us; skew 500 ppm
Jun 19 16:35:32 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:32.429224 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:32.430061 1158 webserver.cc:460] Webserver started at http://127.0.98.2:33175/ using document root <none> and password file <none>
I20260619 16:35:32.430779 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:32.430868 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:32.431147 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:32.433137 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "83b2561e378549fba3b54933f8f97a10"
format_stamp: "Formatted at 2026-06-19 16:35:32 on dist-test-slave-0f28"
server_key: "f441dece0a151a39a9c99f175a8dc221"
server_key_iv: "ed09e022922611ce11cbd4aa90554f34"
server_key_version: "encryptionkey@0"
I20260619 16:35:32.433758 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83b2561e378549fba3b54933f8f97a10"
format_stamp: "Formatted at 2026-06-19 16:35:32 on dist-test-slave-0f28"
server_key: "f441dece0a151a39a9c99f175a8dc221"
server_key_iv: "ed09e022922611ce11cbd4aa90554f34"
server_key_version: "encryptionkey@0"
I20260619 16:35:32.437256 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:32.439992 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:32.441422 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 16:35:32.441550 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "83b2561e378549fba3b54933f8f97a10"
format_stamp: "Formatted at 2026-06-19 16:35:32 on dist-test-slave-0f28"
server_key: "f441dece0a151a39a9c99f175a8dc221"
server_key_iv: "ed09e022922611ce11cbd4aa90554f34"
server_key_version: "encryptionkey@0"
I20260619 16:35:32.441663 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:32.456243 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:32.459260 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:32.459466 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:32.460053 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:32.460875 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:32.460952 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:32.461026 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:32.461086 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:32.470588 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45309
I20260619 16:35:32.470607 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45309 every 8 connection(s)
I20260619 16:35:32.471421 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:32.476394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1158
I20260619 16:35:32.476509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:32.476794 392 external_mini_cluster.cc:1428] Setting key de6bf4e4203f301383e3b53d70a7e80b
Jun 19 16:35:32 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:32.487131 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:32.473419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43355 (local address 127.0.98.62:36827)
0619 16:35:32.473717 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:32.473724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:32.474741 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 16:35:32.476073 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:32.476079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:32.476146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:32.476257 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:32.477827 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:32.478548 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:32.479826 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:32.480123 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:32.480205 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:32.482754 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:32.482760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:32.482763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:32.482795 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:32.484679 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:32.485207 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:32.485211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:32.485215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:32.485339 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:32.485787 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:32.485793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:32.485796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:32.486042 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:32.486166 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:32.486781 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 16:35:32.486901 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":106,"threads_started":1}
I20260619 16:35:32.487890 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:32.473841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:43355)
0619 16:35:32.474575 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:32.474628 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:32.475863 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:32.476488 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:32.476499 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:32.477015 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:32.477683 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:32.477699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:32.478681 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:32.478688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:32.479689 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:32.479699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:32.480272 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:32.480277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:32.480490 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:32.481105 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:32.481126 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:32.482587 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:32.484799 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:32.484803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:32.484815 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:32.485085 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:32.485460 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:32.485463 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:32.485465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:32.485656 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:32.486175 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:32.486180 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:32.486411 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 16:35:32.486686 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 16:35:32.486979 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":120,"threads_started":1}
I20260619 16:35:32.489087 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260619 16:35:32.489296 1287 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:32.489786 1287 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260619 16:35:32.490811 962 ts_manager.cc:194] Registered new tserver with Master: 83b2561e378549fba3b54933f8f97a10 (127.0.98.2:45309)
I20260619 16:35:32.491425 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:35:33.493499 1287 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260619 16:35:33.562736 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.084s user 0.002s sys 0.001s
I20260619 16:35:33.563287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:33.674947 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:33.675213 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:33.675284 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:33.679347 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:33.679436 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:33.679528 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:33.685376 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:33.686517 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:33.687592 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:35:33.697733 1293 server_base.cc:1034] running on GCE node
W20260619 16:35:33.697710 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:33.697680 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:33.697664 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:33.698405 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:33.699036 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:33.700238 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781886933700204 us; error 46 us; skew 500 ppm
Jun 19 16:35:33 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:33.703207 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:33.704092 1293 webserver.cc:460] Webserver started at http://127.0.98.3:37685/ using document root <none> and password file <none>
I20260619 16:35:33.704769 1293 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:33.704864 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:33.705089 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:33.707197 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "fd58c9eb760546eba4b7f91eb98cb0a4"
format_stamp: "Formatted at 2026-06-19 16:35:33 on dist-test-slave-0f28"
server_key: "287a02fd05bfc8b7ee10a15418457a2e"
server_key_iv: "118f8fc196334cd7141022de6e340f7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:33.707762 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fd58c9eb760546eba4b7f91eb98cb0a4"
format_stamp: "Formatted at 2026-06-19 16:35:33 on dist-test-slave-0f28"
server_key: "287a02fd05bfc8b7ee10a15418457a2e"
server_key_iv: "118f8fc196334cd7141022de6e340f7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:33.711552 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:35:33.714071 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:33.715325 1293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:33.715451 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "fd58c9eb760546eba4b7f91eb98cb0a4"
format_stamp: "Formatted at 2026-06-19 16:35:33 on dist-test-slave-0f28"
server_key: "287a02fd05bfc8b7ee10a15418457a2e"
server_key_iv: "118f8fc196334cd7141022de6e340f7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:33.715562 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:33.752806 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:33.756285 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:33.756484 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:33.757071 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:33.757939 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:33.758005 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:33.758067 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:33.758126 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:33.767022 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39935
I20260619 16:35:33.767062 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39935 every 8 connection(s)
I20260619 16:35:33.767911 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:33.769652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1293
I20260619 16:35:33.769773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:33.770077 392 external_mini_cluster.cc:1428] Setting key 025028d72f95e29dc43a8b7e326f5004
Jun 19 16:35:33 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:33.782864 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.769758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35053 (local address 127.0.98.62:36827)
0619 16:35:33.770025 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.770031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.770909 (+ 878us) server_negotiation.cc:407] Connection header received
0619 16:35:33.772751 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.772753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.772798 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.772877 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:33.774148 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.774772 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.776197 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.776422 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.776513 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.779219 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:33.779222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:33.779224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:33.779248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:33.780739 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:33.781269 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:33.781274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:33.781276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:33.781418 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:33.781767 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:33.781771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:33.781773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:33.781967 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:33.782079 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.782550 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.782686 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260619 16:35:33.783849 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.770210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:35053)
0619 16:35:33.770738 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.770800 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.772509 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.773016 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.773022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.773410 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:33.774016 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.774030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.774880 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.774883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.776075 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.776085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.776600 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.776607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.776798 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.777422 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:33.777438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:33.779027 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:33.780868 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:33.780875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:33.780890 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:33.781161 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:33.781520 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:33.781523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:33.781525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:33.781669 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:33.782074 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:33.782081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:33.782320 (+ 239us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.782508 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.782834 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260619 16:35:33.784840 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260619 16:35:33.785084 1422 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:33.785576 1422 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260619 16:35:33.786855 962 ts_manager.cc:194] Registered new tserver with Master: fd58c9eb760546eba4b7f91eb98cb0a4 (127.0.98.3:39935)
I20260619 16:35:33.787508 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35053
I20260619 16:35:33.795838 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:35:33.806638 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.799335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.0.98.62:36827)
0619 16:35:33.799489 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.799495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.799763 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:35:33.799848 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.799850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.799890 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.799960 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:33.800685 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.801187 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.802179 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.802393 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.802472 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.803814 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:33.803817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:33.803819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:33.803841 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:33.805139 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:33.805568 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:33.805570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:33.805571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:33.805660 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:33.805887 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:33.805889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:33.805890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:33.806068 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:33.806167 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.806335 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.806426 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:35:33.811437 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:33.814433 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:33.828914 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.822921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39935 (local address 127.0.0.1:50284)
0619 16:35:33.823737 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.823786 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.823928 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.824682 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.824688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.824710 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.824841 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.824847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.826062 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.826068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.827455 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.827467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.828363 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.828367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.828464 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.828496 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.828631 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.828759 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"spinlock_wait_cycles":5120,"thread_start_us":97,"threads_started":1}
I20260619 16:35:33.828902 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.823218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.0.98.3:39935)
0619 16:35:33.824199 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.824212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.824239 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:35:33.824313 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.824320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.824520 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.824703 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.824961 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.825942 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.827593 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.828243 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.828363 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.828501 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.828663 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.828749 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":887,"thread_start_us":139,"threads_started":1}
I20260619 16:35:33.829428 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.823313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40065 (local address 127.0.0.1:50028)
0619 16:35:33.823736 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.823783 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.823926 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.824829 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.824832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.824854 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.824965 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.824972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.826488 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.826492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.827662 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.827668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.828888 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.828892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.829016 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.829050 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.829118 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.829210 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":70,"spinlock_wait_cycles":25472,"thread_start_us":74,"threads_started":1}
I20260619 16:35:33.829463 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.823990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.0.98.2:45309)
0619 16:35:33.824499 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.824505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.824537 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:35:33.824596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.824601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.824804 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.824934 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.825098 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.826314 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.827853 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.828606 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.828699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.828825 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.829210 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.829306 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":84,"threads_started":1}
I20260619 16:35:33.829450 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.823619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45309 (local address 127.0.0.1:40482)
0619 16:35:33.824314 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.824327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.824401 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.824909 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.824914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.824932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.825011 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.825016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.826623 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.826626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.827726 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.827736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.828888 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.828892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.829021 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.829056 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.829118 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.829258 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":117,"thread_start_us":74,"threads_started":1}
I20260619 16:35:33.832429 1356 tablet_service.cc:1467] Processing CreateTablet for tablet d2c6fe38c4a340a5bf55d73bdc917b42 (DEFAULT_TABLE table=test-table [id=27657e55812f4d7e853b6ab4de4e543a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:33.832653 1220 tablet_service.cc:1467] Processing CreateTablet for tablet d2c6fe38c4a340a5bf55d73bdc917b42 (DEFAULT_TABLE table=test-table [id=27657e55812f4d7e853b6ab4de4e543a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:33.833271 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.823466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.0.98.1:40065)
0619 16:35:33.824393 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.824401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.824424 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:35:33.824494 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.824499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.824685 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.825116 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.825283 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.826398 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.827767 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.828578 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.828699 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.828825 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.830148 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.833142 (+ 2994us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":96,"threads_started":1}
I20260619 16:35:33.833545 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c6fe38c4a340a5bf55d73bdc917b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:33.833606 1220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c6fe38c4a340a5bf55d73bdc917b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:33.835732 1087 tablet_service.cc:1467] Processing CreateTablet for tablet d2c6fe38c4a340a5bf55d73bdc917b42 (DEFAULT_TABLE table=test-table [id=27657e55812f4d7e853b6ab4de4e543a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:33.836678 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c6fe38c4a340a5bf55d73bdc917b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:33.839859 1441 tablet_bootstrap.cc:492] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: Bootstrap starting.
I20260619 16:35:33.842458 1441 tablet_bootstrap.cc:654] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:33.843174 1441 log.cc:824] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:33.844866 1441 tablet_bootstrap.cc:492] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: No bootstrap required, opened a new log
I20260619 16:35:33.845069 1441 ts_tablet_manager.cc:1397] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:35:33.845134 1444 tablet_bootstrap.cc:492] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: Bootstrap starting.
I20260619 16:35:33.845372 1443 tablet_bootstrap.cc:492] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: Bootstrap starting.
I20260619 16:35:33.847084 1444 tablet_bootstrap.cc:654] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:33.847821 1443 tablet_bootstrap.cc:654] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:33.848290 1444 log.cc:824] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:33.848996 1443 log.cc:824] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:33.850122 1444 tablet_bootstrap.cc:492] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: No bootstrap required, opened a new log
I20260619 16:35:33.850327 1444 ts_tablet_manager.cc:1397] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:35:33.850214 1441 raft_consensus.cc:348] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.850474 1441 raft_consensus.cc:374] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:33.850586 1441 raft_consensus.cc:729] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b2561e378549fba3b54933f8f97a10, State: Initialized, Role: FOLLOWER
I20260619 16:35:33.850864 1443 tablet_bootstrap.cc:492] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: No bootstrap required, opened a new log
I20260619 16:35:33.851023 1443 ts_tablet_manager.cc:1397] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:35:33.851058 1441 consensus_queue.cc:260] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.852049 1441 ts_tablet_manager.cc:1428] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:35:33.854069 1444 raft_consensus.cc:348] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.854353 1444 raft_consensus.cc:374] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:33.854476 1444 raft_consensus.cc:729] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd58c9eb760546eba4b7f91eb98cb0a4, State: Initialized, Role: FOLLOWER
I20260619 16:35:33.854575 1443 raft_consensus.cc:348] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.854825 1443 raft_consensus.cc:374] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:33.854887 1443 raft_consensus.cc:729] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2709ef5060e14df2bdd594a4b710c6f2, State: Initialized, Role: FOLLOWER
I20260619 16:35:33.854972 1444 consensus_queue.cc:260] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.855279 1443 consensus_queue.cc:260] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.856077 1443 ts_tablet_manager.cc:1428] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:35:33.856778 1444 ts_tablet_manager.cc:1428] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 16:35:33.857118 1422 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
I20260619 16:35:33.947486 1448 raft_consensus.cc:482] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:33.947775 1448 raft_consensus.cc:504] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.948935 1448 leader_election.cc:288] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b2561e378549fba3b54933f8f97a10 (127.0.98.2:45309), 2709ef5060e14df2bdd594a4b710c6f2 (127.0.98.1:40065)
I20260619 16:35:33.952783 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.949160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45309 (local address 127.0.98.3:38629)
0619 16:35:33.949260 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.949275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.949388 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.949757 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.949761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.949783 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.949908 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.949918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.950741 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.950745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.951659 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.951674 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.952340 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.952343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.952450 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.952475 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.952561 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.952629 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":21}
I20260619 16:35:33.952833 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.949395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38629 (local address 127.0.98.2:45309)
0619 16:35:33.949525 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.949529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.949543 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:35:33.949593 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.949596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.949651 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.949723 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.950024 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.950631 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.951810 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.952240 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.952316 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.952365 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.952593 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.952662 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:35:33.953091 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.949160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40065 (local address 127.0.98.3:46459)
0619 16:35:33.949623 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.949635 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.949743 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.950015 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.950018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.950033 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.950142 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.950148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.950854 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.950857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.951679 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.951686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.952604 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.952608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.952705 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.952727 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.952843 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.952934 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":79,"thread_start_us":170,"threads_started":1}
I20260619 16:35:33.953118 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.949270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46459 (local address 127.0.98.1:40065)
0619 16:35:33.949458 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.949463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.949727 (+ 264us) server_negotiation.cc:407] Connection header received
0619 16:35:33.949827 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.949830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.949884 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.949990 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.950251 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.950746 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.951959 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.952480 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.952593 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.952647 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.952904 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.953006 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 16:35:33.953562 1447 raft_consensus.cc:482] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:33.953617 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b2561e378549fba3b54933f8f97a10" is_pre_election: true
I20260619 16:35:33.953743 1447 raft_consensus.cc:504] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:33.953855 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2709ef5060e14df2bdd594a4b710c6f2" is_pre_election: true
I20260619 16:35:33.954133 1107 raft_consensus.cc:2454] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd58c9eb760546eba4b7f91eb98cb0a4 in term 0.
I20260619 16:35:33.954145 1241 raft_consensus.cc:2454] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd58c9eb760546eba4b7f91eb98cb0a4 in term 0.
I20260619 16:35:33.954610 1312 leader_election.cc:302] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2709ef5060e14df2bdd594a4b710c6f2, 83b2561e378549fba3b54933f8f97a10, fd58c9eb760546eba4b7f91eb98cb0a4; no voters:
I20260619 16:35:33.954890 1447 leader_election.cc:288] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd58c9eb760546eba4b7f91eb98cb0a4 (127.0.98.3:39935), 2709ef5060e14df2bdd594a4b710c6f2 (127.0.98.1:40065)
I20260619 16:35:33.955068 1448 raft_consensus.cc:2781] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:33.955156 1448 raft_consensus.cc:482] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:33.955307 1448 raft_consensus.cc:3037] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:33.958647 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.955080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40065 (local address 127.0.98.2:39007)
0619 16:35:33.955465 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.955478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.955544 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.955873 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.955879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.955897 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.956007 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.956012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.956679 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.956683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.957619 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.957627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.958235 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.958238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.958340 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.958387 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.958444 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.958490 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260619 16:35:33.958659 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.955421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39007 (local address 127.0.98.1:40065)
0619 16:35:33.955516 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.955522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.955534 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:35:33.955658 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.955670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.955713 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.955761 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.956113 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.956561 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.957749 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.958117 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.958229 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.958281 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.958485 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.958555 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:35:33.958736 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.955395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59851 (local address 127.0.98.3:39935)
0619 16:35:33.955494 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:33.955498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:33.955510 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:35:33.955551 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:33.955553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:33.955595 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:33.955672 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:33.955949 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.956591 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.957723 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.958115 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.958238 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.958287 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:33.958556 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 16:35:33.958617 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:35:33.958725 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:33.955019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39935 (local address 127.0.98.2:59851)
0619 16:35:33.955337 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:33.955352 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:33.955440 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:33.955679 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:33.955682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:33.955695 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:33.955835 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.955842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.956720 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.956727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.957592 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:33.957600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:33.958245 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:33.958249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:33.958408 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:33.958445 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:35:33.958523 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:35:33.958584 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":46,"thread_start_us":153,"threads_started":1}
I20260619 16:35:33.959251 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "83b2561e378549fba3b54933f8f97a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2709ef5060e14df2bdd594a4b710c6f2" is_pre_election: true
I20260619 16:35:33.959393 1107 raft_consensus.cc:2454] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b2561e378549fba3b54933f8f97a10 in term 0.
I20260619 16:35:33.959501 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "83b2561e378549fba3b54933f8f97a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" is_pre_election: true
I20260619 16:35:33.959734 1177 leader_election.cc:302] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2709ef5060e14df2bdd594a4b710c6f2, 83b2561e378549fba3b54933f8f97a10; no voters:
I20260619 16:35:33.959983 1447 raft_consensus.cc:2781] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:33.960049 1447 raft_consensus.cc:482] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:33.960079 1447 raft_consensus.cc:3037] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 0 FOLLOWER]: Advancing to term 1
W20260619 16:35:33.973465 1288 tablet.cc:2369] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:34.006014 1448 raft_consensus.cc:504] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:34.006014 1447 raft_consensus.cc:504] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:34.006525 1447 leader_election.cc:288] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [CANDIDATE]: Term 1 election: Requested vote from peers fd58c9eb760546eba4b7f91eb98cb0a4 (127.0.98.3:39935), 2709ef5060e14df2bdd594a4b710c6f2 (127.0.98.1:40065)
I20260619 16:35:34.006978 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "83b2561e378549fba3b54933f8f97a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4"
I20260619 16:35:34.007117 1448 leader_election.cc:288] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [CANDIDATE]: Term 1 election: Requested vote from peers 83b2561e378549fba3b54933f8f97a10 (127.0.98.2:45309), 2709ef5060e14df2bdd594a4b710c6f2 (127.0.98.1:40065)
I20260619 16:35:34.007009 1376 raft_consensus.cc:2379] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b2561e378549fba3b54933f8f97a10 in current term 1: Already voted for candidate fd58c9eb760546eba4b7f91eb98cb0a4 in this term.
I20260619 16:35:34.007174 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b2561e378549fba3b54933f8f97a10"
I20260619 16:35:34.007387 1241 raft_consensus.cc:2379] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fd58c9eb760546eba4b7f91eb98cb0a4 in current term 1: Already voted for candidate 83b2561e378549fba3b54933f8f97a10 in this term.
I20260619 16:35:34.007768 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "83b2561e378549fba3b54933f8f97a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2709ef5060e14df2bdd594a4b710c6f2"
I20260619 16:35:34.007915 1107 raft_consensus.cc:3037] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:34.008276 1106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c6fe38c4a340a5bf55d73bdc917b42" candidate_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2709ef5060e14df2bdd594a4b710c6f2"
I20260619 16:35:34.008767 1312 leader_election.cc:302] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd58c9eb760546eba4b7f91eb98cb0a4; no voters: 2709ef5060e14df2bdd594a4b710c6f2, 83b2561e378549fba3b54933f8f97a10
I20260619 16:35:34.009011 1448 raft_consensus.cc:2726] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 16:35:34.009645 1107 raft_consensus.cc:2454] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b2561e378549fba3b54933f8f97a10 in term 1.
I20260619 16:35:34.009950 1177 leader_election.cc:302] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2709ef5060e14df2bdd594a4b710c6f2, 83b2561e378549fba3b54933f8f97a10; no voters: fd58c9eb760546eba4b7f91eb98cb0a4
I20260619 16:35:34.010175 1447 raft_consensus.cc:2781] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:34.010427 1447 raft_consensus.cc:686] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [term 1 LEADER]: Becoming Leader. State: Replica: 83b2561e378549fba3b54933f8f97a10, State: Running, Role: LEADER
I20260619 16:35:34.010779 1447 consensus_queue.cc:237] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } }
I20260619 16:35:34.014269 962 catalog_manager.cc:5495] T d2c6fe38c4a340a5bf55d73bdc917b42 P 83b2561e378549fba3b54933f8f97a10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b2561e378549fba3b54933f8f97a10 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "83b2561e378549fba3b54933f8f97a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd58c9eb760546eba4b7f91eb98cb0a4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b2561e378549fba3b54933f8f97a10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2709ef5060e14df2bdd594a4b710c6f2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40065 } health_report { overall_health: UNKNOWN } } }
W20260619 16:35:34.018810 1423 tablet.cc:2369] T d2c6fe38c4a340a5bf55d73bdc917b42 P fd58c9eb760546eba4b7f91eb98cb0a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 16:35:34 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260619 16:35:34.067857 1154 tablet.cc:2369] T d2c6fe38c4a340a5bf55d73bdc917b42 P 2709ef5060e14df2bdd594a4b710c6f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:34.070812 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:34.061442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50044 (local address 127.0.98.1:40065)
0619 16:35:34.061642 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:34.061649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:34.062088 (+ 439us) server_negotiation.cc:407] Connection header received
0619 16:35:34.062251 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:34.062256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:34.062318 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:34.062418 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:34.063092 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:34.063664 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:34.064532 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:34.064753 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:34.064864 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:34.067325 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:34.067346 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:34.067361 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:34.067382 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:34.069052 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:34.069541 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:34.069554 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:34.069555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:34.069627 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:34.069925 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:34.069928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:34.069930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:34.070158 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:34.070376 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:34.070537 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 16:35:34.070654 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 16:35:34 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:35:34 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:34.108052 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:34.098878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.0.98.1:40065)
0619 16:35:34.099009 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:34.099013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:34.099044 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:35:34.099093 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:34.099095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:34.099135 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:34.099228 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:34.100116 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:34.100663 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:34.102075 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:34.102304 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:34.102390 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:34.104816 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:34.104819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:34.104820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:34.104843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:34.106335 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:34.106782 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:34.106785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:34.106786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:34.106861 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:34.107225 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:34.107228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:34.107230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:34.107397 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:34.107499 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:34.107781 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 16:35:34.107874 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:35:34 dist-test-slave-0f28 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:35:34 dist-test-slave-0f28 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:34.212153 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:34.202349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.0.98.1:40065)
0619 16:35:34.202527 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:34.202535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:34.202658 (+ 123us) server_negotiation.cc:407] Connection header received
0619 16:35:34.202770 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:34.202777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:34.202843 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:34.202960 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:34.203788 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:34.204738 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:34.206014 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:34.206247 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:34.206337 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:34.209022 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:34.209025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:34.209027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:34.209050 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:34.210380 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:34.210788 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:34.210791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:34.210792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:34.210866 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:34.211142 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:34.211146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:34.211151 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:34.211335 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:34.211485 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:34.211804 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 16:35:34.211966 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 16:35:34.213752 1055 tablet_service.cc:2908] Rejecting scan request for tablet d2c6fe38c4a340a5bf55d73bdc917b42: Uninitialized: clean time has not yet been initialized
I20260619 16:35:34.215798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1024
I20260619 16:35:34.222551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1158
I20260619 16:35:34.228247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1293
I20260619 16:35:34.233430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 930
2026-06-19T16:35:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5012 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:35 dist-test-slave-0f28 krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:35 dist-test-slave-0f28 krb5kdc[1467](info): set up 2 sockets
Jun 19 16:35:35 dist-test-slave-0f28 krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260619 16:35:36.713001 392 mini_kdc.cc:121] Time spent starting KDC: real 2.456s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:36 dist-test-slave-0f28 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:36.832737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33235
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:36.940595 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:36.940876 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:36.940944 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:36.944849 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:36.944927 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:35:36.944959 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:36.945022 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:36.945060 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:36.945091 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:36.950181 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33235
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:36.951145 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:36.952103 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:36.960044 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:36.960067 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:36.960053 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:36.960316 1480 server_base.cc:1034] running on GCE node
I20260619 16:35:36.960934 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:36.962047 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:36.963239 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781886936963206 us; error 41 us; skew 500 ppm
Jun 19 16:35:36 dist-test-slave-0f28 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:36.965833 1480 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:36.966609 1480 webserver.cc:460] Webserver started at http://127.0.98.62:35035/ using document root <none> and password file <none>
I20260619 16:35:36.967121 1480 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:36.967206 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:36.967446 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:36.969394 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "3911884113f24b4397be2da7baddf89c"
format_stamp: "Formatted at 2026-06-19 16:35:36 on dist-test-slave-0f28"
server_key: "bcf71ace61af4401e0b3c9d1e052f537"
server_key_iv: "c189d071938f12b362f4488e85489c7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:36.969946 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "3911884113f24b4397be2da7baddf89c"
format_stamp: "Formatted at 2026-06-19 16:35:36 on dist-test-slave-0f28"
server_key: "bcf71ace61af4401e0b3c9d1e052f537"
server_key_iv: "c189d071938f12b362f4488e85489c7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:36.973387 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:35:36.975631 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:36.976723 1480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:35:36.976853 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "3911884113f24b4397be2da7baddf89c"
format_stamp: "Formatted at 2026-06-19 16:35:36 on dist-test-slave-0f28"
server_key: "bcf71ace61af4401e0b3c9d1e052f537"
server_key_iv: "c189d071938f12b362f4488e85489c7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:36.976956 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:36.997102 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:37.000252 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:37.000444 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:37.007902 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33235
I20260619 16:35:37.007910 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33235 every 8 connection(s)
I20260619 16:35:37.008771 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:37.011394 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:37.016418 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: Bootstrap starting.
I20260619 16:35:37.018090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1480
I20260619 16:35:37.018198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:37.018446 392 external_mini_cluster.cc:1428] Setting key 96dd30e44b856e2bca99e3fbca78df1d
I20260619 16:35:37.018960 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:37.019848 1558 log.cc:824] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:37.021821 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: No bootstrap required, opened a new log
I20260619 16:35:37.025692 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3911884113f24b4397be2da7baddf89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260619 16:35:37.025964 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:37.026029 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3911884113f24b4397be2da7baddf89c, State: Initialized, Role: FOLLOWER
I20260619 16:35:37.026532 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3911884113f24b4397be2da7baddf89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260619 16:35:37.026737 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:37.026808 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:37.026922 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:37.027792 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3911884113f24b4397be2da7baddf89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260619 16:35:37.028107 1558 leader_election.cc:302] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3911884113f24b4397be2da7baddf89c; no voters:
I20260619 16:35:37.028450 1558 leader_election.cc:288] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:37.028555 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:37.028784 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [term 1 LEADER]: Becoming Leader. State: Replica: 3911884113f24b4397be2da7baddf89c, State: Running, Role: LEADER
I20260619 16:35:37.029088 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3911884113f24b4397be2da7baddf89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } }
I20260619 16:35:37.029670 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:37.030761 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3911884113f24b4397be2da7baddf89c. Latest consensus state: current_term: 1 leader_uuid: "3911884113f24b4397be2da7baddf89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3911884113f24b4397be2da7baddf89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } } }
I20260619 16:35:37.030879 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:37.031185 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3911884113f24b4397be2da7baddf89c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3911884113f24b4397be2da7baddf89c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33235 } } }
I20260619 16:35:37.031276 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:37.033713 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.019507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.0.98.62:33235)
0619 16:35:37.019890 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.019935 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.019976 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:35:37.020750 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.020768 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.021062 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.021370 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.022201 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.023177 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.024403 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.024905 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.025027 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.027836 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.027841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.027853 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.027906 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.029969 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.030506 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.030515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.030523 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.030684 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.030997 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.031003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.031008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.031552 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.031768 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.032469 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.032773 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
W20260619 16:35:37.034587 1578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:35:37.034673 1578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:35:37.034768 1572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:37.035753 1572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:37.040985 1572 catalog_manager.cc:1347] Generated new cluster ID: 6a1110e1c2344e8d95b0605f6c15c2da
I20260619 16:35:37.041065 1572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:37.053342 1572 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:37.054214 1572 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:37.057741 1572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: Generated new TSK 0
I20260619 16:35:37.058363 1572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:37.086481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:37.220676 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:37.220908 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:37.220986 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:37.224979 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:37.225030 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:37.225068 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:35:37.225159 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:37.230360 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:37.231405 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:37.232414 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:37.241516 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:37.241576 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:37.241521 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:37.241868 1584 server_base.cc:1034] running on GCE node
I20260619 16:35:37.242321 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:37.242973 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:37.244161 1584 hybrid_clock.cc:648] HybridClock initialized: now 1781886937244144 us; error 29 us; skew 500 ppm
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:37.246869 1584 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:37.247645 1584 webserver.cc:460] Webserver started at http://127.0.98.1:43189/ using document root <none> and password file <none>
I20260619 16:35:37.248176 1584 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:37.248265 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:37.248479 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:37.250402 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "720a24cf55884c4cb7ba2c83f901ca7d"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "cbcc451dbf8c4c6cd480cafebebafbe6"
server_key_iv: "452e2544d756f81c3bb14e0bef51e872"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.250944 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "720a24cf55884c4cb7ba2c83f901ca7d"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "cbcc451dbf8c4c6cd480cafebebafbe6"
server_key_iv: "452e2544d756f81c3bb14e0bef51e872"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.254388 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:35:37.257035 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.258252 1584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:35:37.258380 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "720a24cf55884c4cb7ba2c83f901ca7d"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "cbcc451dbf8c4c6cd480cafebebafbe6"
server_key_iv: "452e2544d756f81c3bb14e0bef51e872"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.258494 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:37.279362 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:37.282270 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:37.282406 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:37.283828 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:37.283921 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.284003 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:37.284053 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.295464 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43585
I20260619 16:35:37.295531 1714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43585 every 8 connection(s)
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:37.296391 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:37.299852 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.288107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57299 (local address 127.0.98.62:33235)
0619 16:35:37.288231 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.288244 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.288261 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:35:37.288312 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.288316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.288372 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.288451 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.290134 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.291033 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.292860 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.293271 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.293393 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.296461 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.296464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.296467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.296493 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.298071 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.298545 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.298548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.298551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.298634 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.298932 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.298935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.298936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.299121 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.299216 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.299636 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.299731 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:35:37.300626 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.285746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.1:57299)
0619 16:35:37.286372 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.286419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.287740 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.288748 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.288758 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.289233 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:37.289947 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.289969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.291205 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.291209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.292718 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.292728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.293490 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.293493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.293825 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.294525 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:37.294545 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:37.296283 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:37.298189 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.298193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.298203 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.298457 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.298720 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.298722 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.298723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.298848 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.299221 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:37.299225 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:37.299434 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.299581 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.299810 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260619 16:35:37.301889 1715 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33235
I20260619 16:35:37.302029 1715 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:37.302484 1715 heartbeater.cc:511] Master 127.0.98.62:33235 requested a full tablet report, sending...
I20260619 16:35:37.302767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1584
I20260619 16:35:37.302861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:37.303124 392 external_mini_cluster.cc:1428] Setting key e1e66f3795a66646feaae0d49490d1cc
I20260619 16:35:37.304121 1511 ts_manager.cc:194] Registered new tserver with Master: 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585)
I20260619 16:35:37.305583 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57299
I20260619 16:35:37.311527 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.304069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59093 (local address 127.0.98.62:33235)
0619 16:35:37.304173 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.304178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.304465 (+ 287us) server_negotiation.cc:407] Connection header received
0619 16:35:37.304634 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.304637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.304680 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.304770 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.305231 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.306064 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.307114 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.307374 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.307495 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.308541 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.308544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.308547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.308569 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.309979 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.310394 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.310399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.310401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.310476 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.310748 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.310753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.310755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.310926 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.311038 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.311282 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.311387 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:35:37.311551 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.303995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.1:59093)
0619 16:35:37.304392 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.304407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.304521 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.304815 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.304818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.305007 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:37.305114 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.305120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.306191 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.306194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.306984 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.306995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.307499 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.307502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.307610 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.308091 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:37.308103 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:37.308401 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:37.310107 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.310110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.310112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.310298 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.310570 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.310574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.310576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.310661 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.311022 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:37.311025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:37.311112 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.311223 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.311353 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":4,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:37.328558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:37.433064 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:37.433287 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:37.433401 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:37.437242 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:37.437294 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:37.437384 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:35:37.437441 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:37.442329 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:37.443312 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:37.444303 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:37.453684 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:37.453676 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:37.453837 1724 server_base.cc:1034] running on GCE node
W20260619 16:35:37.453675 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:37.454344 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:37.454916 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:37.456084 1724 hybrid_clock.cc:648] HybridClock initialized: now 1781886937456071 us; error 24 us; skew 500 ppm
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:37.461968 1724 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:37.462831 1724 webserver.cc:460] Webserver started at http://127.0.98.2:37989/ using document root <none> and password file <none>
I20260619 16:35:37.463384 1724 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:37.463474 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:37.463693 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:37.465689 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "08dbc9d0624a4fbbb184c39f776dff2c"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "c5ea079e4f41ad1af4b837b2ec85458e"
server_key_iv: "25557ab363ebe02a20f7ef79b707bc43"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.466257 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08dbc9d0624a4fbbb184c39f776dff2c"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "c5ea079e4f41ad1af4b837b2ec85458e"
server_key_iv: "25557ab363ebe02a20f7ef79b707bc43"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.469769 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:37.472047 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.473289 1724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:37.473482 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "08dbc9d0624a4fbbb184c39f776dff2c"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "c5ea079e4f41ad1af4b837b2ec85458e"
server_key_iv: "25557ab363ebe02a20f7ef79b707bc43"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.473606 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:37.493887 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:37.496658 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:37.496861 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:37.498219 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:37.498293 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.498360 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:37.498411 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:37.508968 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42289
I20260619 16:35:37.509013 1854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42289 every 8 connection(s)
I20260619 16:35:37.509882 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:37.512315 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.499083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60235 (local address 127.0.98.62:33235)
0619 16:35:37.499209 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.499213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.500233 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 16:35:37.501416 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.501419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.501476 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.501601 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.503211 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.504138 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.505434 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.505666 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.505782 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.508723 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.508727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.508730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.508755 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.510331 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.510871 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.510876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.510878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.510963 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.511293 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.511297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.511299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.511473 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.511581 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.512035 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.512149 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:35:37.512985 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.499582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.2:60235)
0619 16:35:37.500077 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.500116 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.501147 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.501802 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.501812 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.502299 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:37.503057 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.503080 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.504253 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.504260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.505274 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.505306 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.505827 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.505831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.505998 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.507118 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:37.507133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:37.508494 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:37.510458 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.510463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.510475 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.510766 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.511063 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.511067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.511069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.511196 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.511578 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:37.511585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:37.511803 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.511984 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.512263 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260619 16:35:37.514035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1724
I20260619 16:35:37.514160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:37.514202 1855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33235
I20260619 16:35:37.514451 392 external_mini_cluster.cc:1428] Setting key efc02db4656b8730de921d98c6af6fa4
I20260619 16:35:37.514914 1855 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:37.515471 1855 heartbeater.cc:511] Master 127.0.98.62:33235 requested a full tablet report, sending...
I20260619 16:35:37.516722 1512 ts_manager.cc:194] Registered new tserver with Master: 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2:42289)
I20260619 16:35:37.520244 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60235
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:37.531253 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.519052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42619 (local address 127.0.98.62:33235)
0619 16:35:37.519388 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.519394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.521347 (+ 1953us) server_negotiation.cc:407] Connection header received
0619 16:35:37.521520 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.521526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.521575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.521627 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.522470 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.523222 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.524727 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.525303 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.525432 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.527040 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.527044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.527047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.527073 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.528949 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.529509 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.529526 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.529529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.529633 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.530009 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.530012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.530014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.530291 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.530372 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.530972 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.531090 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260619 16:35:37.531720 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.516862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.2:42619)
0619 16:35:37.521252 (+ 4390us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.521269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.521418 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.521785 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.521791 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.522023 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:37.522226 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.522260 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.523355 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.523358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.524496 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.524510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.525552 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.525555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.525645 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.526125 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:37.526152 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:37.526869 (+ 717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:37.529085 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.529090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.529092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.529379 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.529800 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.529805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.529807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.529903 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.530456 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:37.530459 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:37.530695 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.531398 (+ 703us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.531574 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4283,"thread_start_us":130,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:37.553120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33235
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:37.662912 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:37.663126 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:37.663166 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:37.667004 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:37.667052 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:37.667089 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:35:37.667120 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:37.672070 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:37.672904 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:37.673849 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:37.682801 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:37.682857 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:37.682823 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:37.683092 1864 server_base.cc:1034] running on GCE node
I20260619 16:35:37.683467 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:37.684062 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:37.685213 1864 hybrid_clock.cc:648] HybridClock initialized: now 1781886937685182 us; error 45 us; skew 500 ppm
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:37.687805 1864 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:37.688629 1864 webserver.cc:460] Webserver started at http://127.0.98.3:36131/ using document root <none> and password file <none>
I20260619 16:35:37.689141 1864 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:37.689226 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:37.689474 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:37.691336 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "714b61cb367d456f976b7b526a410367"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "e8dbb7737e96252f8f124a3507a51d88"
server_key_iv: "f463edb288bd3212a358522482394d0b"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.691879 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "714b61cb367d456f976b7b526a410367"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "e8dbb7737e96252f8f124a3507a51d88"
server_key_iv: "f463edb288bd3212a358522482394d0b"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.695364 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:37.697755 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.698875 1864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:37.699007 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "714b61cb367d456f976b7b526a410367"
format_stamp: "Formatted at 2026-06-19 16:35:37 on dist-test-slave-0f28"
server_key: "e8dbb7737e96252f8f124a3507a51d88"
server_key_iv: "f463edb288bd3212a358522482394d0b"
server_key_version: "encryptionkey@0"
I20260619 16:35:37.699113 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:37.723335 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:37.726075 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:37.726271 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:37.727622 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:37.727697 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:37.727792 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:37.727837 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:37.739351 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33749
I20260619 16:35:37.739423 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33749 every 8 connection(s)
I20260619 16:35:37.740355 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:37.741567 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.728497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56623 (local address 127.0.98.62:33235)
0619 16:35:37.728668 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.728674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.729523 (+ 849us) server_negotiation.cc:407] Connection header received
0619 16:35:37.730482 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.730485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.730541 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.730630 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.731955 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.732640 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.733828 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.734058 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.734175 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.737853 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.737857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.737861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.737888 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.739534 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.740033 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.740040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.740042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.740125 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.740454 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.740457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.740459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.740679 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.740786 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.741313 (+ 527us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.741454 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 16:35:37.742287 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.728875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.3:56623)
0619 16:35:37.729373 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.729421 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.730267 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.730918 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.730928 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.731260 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:37.731821 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.731834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.732758 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.732762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.733711 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.733720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.734209 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.734213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.734438 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.735036 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:37.735060 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:37.737708 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:37.739655 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.739659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.739669 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.739914 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.740221 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.740223 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.740225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.740345 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.740791 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:37.740796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:37.741053 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.741235 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.741510 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":139,"threads_started":1}
I20260619 16:35:37.743400 1995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33235
I20260619 16:35:37.743548 1995 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:37.744066 1995 heartbeater.cc:511] Master 127.0.98.62:33235 requested a full tablet report, sending...
I20260619 16:35:37.745013 1511 ts_manager.cc:194] Registered new tserver with Master: 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749)
I20260619 16:35:37.745755 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56623
I20260619 16:35:37.749447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 1864
I20260619 16:35:37.749576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:37.749852 392 external_mini_cluster.cc:1428] Setting key c2f19d5954bc0f05a538601f2d8f37a2
I20260619 16:35:37.751648 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:35:37.752219 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.746023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37679 (local address 127.0.98.62:33235)
0619 16:35:37.746128 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.746132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.746144 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:35:37.746176 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.746177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.746209 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.746298 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.746680 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.747166 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.747948 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.748171 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.748272 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.749171 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.749174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.749177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.749200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.750710 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.751113 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.751118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.751120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.751197 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.751470 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.751475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.751476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.751634 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.751745 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.751978 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.752074 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:35:37.752250 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.745435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.98.3:37679)
0619 16:35:37.745672 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.745684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.745757 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.746313 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.746316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.746486 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:37.746571 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.746576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.747288 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.747296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.747834 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.747841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.748285 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.748288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.748385 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.748760 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:37.748770 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:37.749043 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:37.750821 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.750828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.750830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.751017 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.751292 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:37.751297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:37.751299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:37.751382 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:37.751727 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:37.751730 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:37.751816 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.751929 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.752060 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:35:37 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:37.776553 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.768581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.98.62:33235)
0619 16:35:37.768746 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.768750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.768761 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:35:37.768791 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.768793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.768827 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.769018 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:37.769736 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.770233 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.771079 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.771291 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.771398 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.773697 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:37.773700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:37.773702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:37.773722 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:37.775118 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.775533 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.775536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.775537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.775609 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:37.775854 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:37.775856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:37.775857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:37.776006 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:37.776099 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.776311 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.776394 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260619 16:35:37.779325 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:37.782248 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:37.796603 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.790363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33749 (local address 127.0.0.1:44834)
0619 16:35:37.791273 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.791286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.791376 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.792197 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.792203 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.792223 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:37.792348 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.792354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.793916 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.793919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.795264 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.795272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.796263 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.796266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.796344 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.796363 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.796446 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.796500 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":53,"thread_start_us":96,"threads_started":1}
I20260619 16:35:37.796691 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.791431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.0.98.3:33749)
0619 16:35:37.791717 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.791728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.791752 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:35:37.791825 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.791830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.792028 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.792189 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:37.792467 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.793694 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.795386 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.796121 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.796228 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.796378 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.796474 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.796555 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
I20260619 16:35:37.796579 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.790284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42289 (local address 127.0.0.1:58690)
0619 16:35:37.790610 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.790653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.790749 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.792336 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.792339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.792356 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:37.792507 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.792515 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.793875 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.793877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.794986 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.794995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.795994 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.795997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.796077 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.796106 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.796251 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.796360 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":139,"spinlock_wait_cycles":2944,"thread_start_us":80,"threads_started":1}
I20260619 16:35:37.797484 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.790957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.0.98.2:42289)
0619 16:35:37.791855 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.791865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.791883 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:35:37.791941 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.791945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.792087 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.792205 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:37.792638 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.793752 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.795108 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.795856 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.795967 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.796116 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.797217 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.797352 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":105,"threads_started":1}
I20260619 16:35:37.798177 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.790985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.0.1:44124)
0619 16:35:37.791665 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.791675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.791785 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.792353 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.792355 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.792370 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:37.792499 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.792504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.793499 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.793504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.795262 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.795272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.797812 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.797816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.797912 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.797933 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.797997 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.798058 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":37,"thread_start_us":84,"threads_started":1}
I20260619 16:35:37.798652 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.791188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.0.98.1:43585)
0619 16:35:37.791552 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.791565 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.791762 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:35:37.791923 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.791928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.792073 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.792233 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:37.792610 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.793382 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.796191 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.797358 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.797582 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.798378 (+ 796us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.798432 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.798525 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":83,"threads_started":1}
I20260619 16:35:37.799804 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 76a9b34198df425bb1484f2b64b3b883 (DEFAULT_TABLE table=test-table [id=c211a112bda34ddd8f6b94e10808ecd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:37.799994 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 76a9b34198df425bb1484f2b64b3b883 (DEFAULT_TABLE table=test-table [id=c211a112bda34ddd8f6b94e10808ecd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:37.800761 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a9b34198df425bb1484f2b64b3b883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:37.800959 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a9b34198df425bb1484f2b64b3b883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:37.801088 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 76a9b34198df425bb1484f2b64b3b883 (DEFAULT_TABLE table=test-table [id=c211a112bda34ddd8f6b94e10808ecd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:37.801995 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76a9b34198df425bb1484f2b64b3b883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:37.807674 2018 tablet_bootstrap.cc:492] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Bootstrap starting.
I20260619 16:35:37.809142 2020 tablet_bootstrap.cc:492] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Bootstrap starting.
I20260619 16:35:37.809932 2018 tablet_bootstrap.cc:654] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:37.809918 2019 tablet_bootstrap.cc:492] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Bootstrap starting.
I20260619 16:35:37.810773 2018 log.cc:824] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:37.811290 2020 tablet_bootstrap.cc:654] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:37.811688 2019 tablet_bootstrap.cc:654] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:37.812085 2020 log.cc:824] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:37.812463 2019 log.cc:824] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:37.813366 2018 tablet_bootstrap.cc:492] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: No bootstrap required, opened a new log
I20260619 16:35:37.813560 2018 ts_tablet_manager.cc:1397] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:35:37.813946 2020 tablet_bootstrap.cc:492] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: No bootstrap required, opened a new log
I20260619 16:35:37.814193 2019 tablet_bootstrap.cc:492] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: No bootstrap required, opened a new log
I20260619 16:35:37.814344 2020 ts_tablet_manager.cc:1397] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 16:35:37.814380 2019 ts_tablet_manager.cc:1397] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:35:37.817127 2019 raft_consensus.cc:348] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.817368 2019 raft_consensus.cc:374] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:37.817421 2019 raft_consensus.cc:729] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dbc9d0624a4fbbb184c39f776dff2c, State: Initialized, Role: FOLLOWER
I20260619 16:35:37.817355 2018 raft_consensus.cc:348] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.817595 2018 raft_consensus.cc:374] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:37.817654 2018 raft_consensus.cc:729] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714b61cb367d456f976b7b526a410367, State: Initialized, Role: FOLLOWER
I20260619 16:35:37.817878 2019 consensus_queue.cc:260] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.817875 2020 raft_consensus.cc:348] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.818073 2020 raft_consensus.cc:374] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:37.818117 2020 raft_consensus.cc:729] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720a24cf55884c4cb7ba2c83f901ca7d, State: Initialized, Role: FOLLOWER
I20260619 16:35:37.818069 2018 consensus_queue.cc:260] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.818523 2020 consensus_queue.cc:260] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.818826 2019 ts_tablet_manager.cc:1428] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:35:37.818966 2018 ts_tablet_manager.cc:1428] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 16:35:37.819205 1855 heartbeater.cc:503] Master 127.0.98.62:33235 was elected leader, sending a full tablet report...
I20260619 16:35:37.819267 1995 heartbeater.cc:503] Master 127.0.98.62:33235 was elected leader, sending a full tablet report...
I20260619 16:35:37.819339 2020 ts_tablet_manager.cc:1428] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260619 16:35:37.820137 1715 heartbeater.cc:503] Master 127.0.98.62:33235 was elected leader, sending a full tablet report...
I20260619 16:35:37.941936 2024 raft_consensus.cc:482] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:37.942153 2024 raft_consensus.cc:504] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.943236 2024 leader_election.cc:288] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749), 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585)
I20260619 16:35:37.947813 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.943596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48777 (local address 127.0.98.3:33749)
0619 16:35:37.943732 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.943737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.943753 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:35:37.943874 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.943880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.944030 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.944127 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:37.944437 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.945085 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.946740 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.947268 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.947371 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.947433 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.947629 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.947703 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:35:37.947862 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.943454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33749 (local address 127.0.98.2:48777)
0619 16:35:37.943632 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.943649 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.943768 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.944140 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.944145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.944165 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:37.944304 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.944313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.945209 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.945211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.946612 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.946623 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.947385 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.947388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.947469 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.947489 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.947615 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.947699 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":21}
I20260619 16:35:37.948117 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.943477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.98.2:38941)
0619 16:35:37.943912 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:37.943924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:37.944053 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:37.944341 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:37.944347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:37.944370 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:37.944812 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.944820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.945623 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.945627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.946860 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:37.946867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.947668 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.947672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:37.947763 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.947786 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:35:37.947869 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:35:37.947942 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":31,"thread_start_us":191,"threads_started":1}
I20260619 16:35:37.948127 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.943698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38941 (local address 127.0.98.1:43585)
0619 16:35:37.943863 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.943867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.944014 (+ 147us) server_negotiation.cc:407] Connection header received
0619 16:35:37.944177 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.944180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.944238 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.944304 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:37.944924 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.945512 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.946962 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.947523 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.947644 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.947698 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.947893 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.947967 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:35:37.948590 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a9b34198df425bb1484f2b64b3b883" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714b61cb367d456f976b7b526a410367" is_pre_election: true
I20260619 16:35:37.948827 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a9b34198df425bb1484f2b64b3b883" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" is_pre_election: true
I20260619 16:35:37.948904 1949 raft_consensus.cc:2454] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 0.
I20260619 16:35:37.949102 1669 raft_consensus.cc:2454] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 0.
I20260619 16:35:37.949469 1742 leader_election.cc:302] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08dbc9d0624a4fbbb184c39f776dff2c, 714b61cb367d456f976b7b526a410367; no voters:
I20260619 16:35:37.949790 2024 raft_consensus.cc:2781] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:37.949878 2024 raft_consensus.cc:482] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:37.949913 2024 raft_consensus.cc:3037] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:37.950915 2024 raft_consensus.cc:504] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.951246 2024 leader_election.cc:288] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 election: Requested vote from peers 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749), 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585)
I20260619 16:35:37.951529 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a9b34198df425bb1484f2b64b3b883" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714b61cb367d456f976b7b526a410367"
I20260619 16:35:37.951545 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76a9b34198df425bb1484f2b64b3b883" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720a24cf55884c4cb7ba2c83f901ca7d"
I20260619 16:35:37.951643 1949 raft_consensus.cc:3037] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:37.951683 1669 raft_consensus.cc:3037] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:37.952661 1669 raft_consensus.cc:2454] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 1.
I20260619 16:35:37.952955 1741 leader_election.cc:302] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08dbc9d0624a4fbbb184c39f776dff2c, 720a24cf55884c4cb7ba2c83f901ca7d; no voters:
I20260619 16:35:37.952993 1949 raft_consensus.cc:2454] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 1.
I20260619 16:35:37.953127 2024 raft_consensus.cc:2781] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:37.953375 2024 raft_consensus.cc:686] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 LEADER]: Becoming Leader. State: Replica: 08dbc9d0624a4fbbb184c39f776dff2c, State: Running, Role: LEADER
I20260619 16:35:37.953722 2024 consensus_queue.cc:237] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } }
I20260619 16:35:37.956820 1510 catalog_manager.cc:5495] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: UNKNOWN } } }
I20260619 16:35:37.986400 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:37.982975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.0.98.2:42289)
0619 16:35:37.983112 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:37.983115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:37.983130 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:35:37.983318 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:37.983321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:37.983369 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:37.983448 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:37.983675 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.984151 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.984989 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:37.985212 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:37.985288 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:37.985627 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:37.985977 (+ 350us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:37.986062 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:37.986190 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 16:35:37.986236 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:35:37.991405 1996 tablet.cc:2369] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:37.991780 1949 raft_consensus.cc:1261] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Refusing update from remote peer 08dbc9d0624a4fbbb184c39f776dff2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:37.991793 1669 raft_consensus.cc:1261] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Refusing update from remote peer 08dbc9d0624a4fbbb184c39f776dff2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:37.992448 2028 consensus_queue.cc:1035] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:37.992678 2024 consensus_queue.cc:1035] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:37.999920 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298608898007908352 to 7298608897860653056. Current Snapshot: MvccSnapshot[applied={T|T < 7298608898007908352}]
I20260619 16:35:38.002249 2033 mvcc.cc:205] Tried to move back new op lower bound from 7298608898007908352 to 7298608897860653056. Current Snapshot: MvccSnapshot[applied={T|T < 7298608898007908352}]
I20260619 16:35:38.010663 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:38.010833 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:38.013104 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: Sending DeleteTablet for 3 replicas of tablet 76a9b34198df425bb1484f2b64b3b883
I20260619 16:35:38.013900 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a9b34198df425bb1484f2b64b3b883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44124
I20260619 16:35:38.013979 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a9b34198df425bb1484f2b64b3b883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44834
I20260619 16:35:38.014379 2041 tablet_replica.cc:330] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: stopping tablet replica
I20260619 16:35:38.014532 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 76a9b34198df425bb1484f2b64b3b883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58690
I20260619 16:35:38.014819 2041 raft_consensus.cc:2229] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:38.015159 2041 raft_consensus.cc:2258] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:38.015878 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:38.016247 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:38.016726 2041 ts_tablet_manager.cc:1905] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:38.020706 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 (DEFAULT_TABLE table=test-table [id=732f6dfb56a74192a461954eb77c839a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:38.020967 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:38.022809 2020 tablet_bootstrap.cc:492] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Bootstrap starting.
I20260619 16:35:38.023298 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 (DEFAULT_TABLE table=test-table [id=732f6dfb56a74192a461954eb77c839a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:38.023540 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:38.023785 2020 tablet_bootstrap.cc:654] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:38.025017 2020 tablet_bootstrap.cc:492] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: No bootstrap required, opened a new log
I20260619 16:35:38.025106 2020 ts_tablet_manager.cc:1397] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 16:35:38.025539 2020 raft_consensus.cc:348] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.025667 2020 raft_consensus.cc:374] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:38.025705 2020 raft_consensus.cc:729] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720a24cf55884c4cb7ba2c83f901ca7d, State: Initialized, Role: FOLLOWER
I20260619 16:35:38.025825 2020 consensus_queue.cc:260] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.026088 2020 ts_tablet_manager.cc:1428] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:38.026151 2018 tablet_bootstrap.cc:492] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Bootstrap starting.
I20260619 16:35:38.027052 2018 tablet_bootstrap.cc:654] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:38.027616 2041 ts_tablet_manager.cc:1918] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:35:38.027711 2041 log.cc:1196] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/76a9b34198df425bb1484f2b64b3b883
I20260619 16:35:38.027745 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 (DEFAULT_TABLE table=test-table [id=732f6dfb56a74192a461954eb77c839a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:38.028389 2041 ts_tablet_manager.cc:1939] T 76a9b34198df425bb1484f2b64b3b883 P 720a24cf55884c4cb7ba2c83f901ca7d: Deleting consensus metadata
I20260619 16:35:38.029384 2018 tablet_bootstrap.cc:492] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: No bootstrap required, opened a new log
I20260619 16:35:38.029469 2018 ts_tablet_manager.cc:1397] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:35:38.029495 2043 tablet_replica.cc:330] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: stopping tablet replica
I20260619 16:35:38.029482 1497 catalog_manager.cc:4841] TS 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585): tablet 76a9b34198df425bb1484f2b64b3b883 (table test-table [id=c211a112bda34ddd8f6b94e10808ecd3]) successfully deleted
I20260619 16:35:38.029778 2043 raft_consensus.cc:2229] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:38.029963 2018 raft_consensus.cc:348] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.030103 2018 raft_consensus.cc:374] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:38.030141 2018 raft_consensus.cc:729] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714b61cb367d456f976b7b526a410367, State: Initialized, Role: FOLLOWER
I20260619 16:35:38.030160 2043 raft_consensus.cc:2258] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:38.030289 2018 consensus_queue.cc:260] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.030496 2042 tablet_replica.cc:330] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: stopping tablet replica
I20260619 16:35:38.030548 2018 ts_tablet_manager.cc:1428] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:38.030748 2042 raft_consensus.cc:2229] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:38.030973 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:38.031047 2042 raft_consensus.cc:2258] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:38.032284 2043 ts_tablet_manager.cc:1905] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:38.033968 2019 tablet_bootstrap.cc:492] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Bootstrap starting.
I20260619 16:35:38.034745 2043 ts_tablet_manager.cc:1918] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:35:38.034843 2043 log.cc:1196] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/76a9b34198df425bb1484f2b64b3b883
I20260619 16:35:38.034866 2019 tablet_bootstrap.cc:654] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:38.035162 2043 ts_tablet_manager.cc:1939] T 76a9b34198df425bb1484f2b64b3b883 P 08dbc9d0624a4fbbb184c39f776dff2c: Deleting consensus metadata
I20260619 16:35:38.035874 2042 ts_tablet_manager.cc:1905] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:38.036334 1497 catalog_manager.cc:4841] TS 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2:42289): tablet 76a9b34198df425bb1484f2b64b3b883 (table test-table [id=c211a112bda34ddd8f6b94e10808ecd3]) successfully deleted
I20260619 16:35:38.038293 2019 tablet_bootstrap.cc:492] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: No bootstrap required, opened a new log
I20260619 16:35:38.038378 2019 ts_tablet_manager.cc:1397] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:35:38.038457 2042 ts_tablet_manager.cc:1918] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:35:38.038546 2042 log.cc:1196] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/76a9b34198df425bb1484f2b64b3b883
I20260619 16:35:38.038853 2042 ts_tablet_manager.cc:1939] T 76a9b34198df425bb1484f2b64b3b883 P 714b61cb367d456f976b7b526a410367: Deleting consensus metadata
I20260619 16:35:38.038859 2019 raft_consensus.cc:348] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.038987 2019 raft_consensus.cc:374] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:38.039031 2019 raft_consensus.cc:729] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dbc9d0624a4fbbb184c39f776dff2c, State: Initialized, Role: FOLLOWER
I20260619 16:35:38.039170 2019 consensus_queue.cc:260] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.039418 2019 ts_tablet_manager.cc:1428] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:38.039904 1498 catalog_manager.cc:4841] TS 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749): tablet 76a9b34198df425bb1484f2b64b3b883 (table test-table [id=c211a112bda34ddd8f6b94e10808ecd3]) successfully deleted
W20260619 16:35:38.047719 1716 tablet.cc:2369] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:38.069370 2038 raft_consensus.cc:482] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:38.069489 2038 raft_consensus.cc:504] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.069939 2038 leader_election.cc:288] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585), 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749)
I20260619 16:35:38.070302 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c55a4d1de434cb77d20e4eb7d89a7" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" is_pre_election: true
I20260619 16:35:38.070329 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c55a4d1de434cb77d20e4eb7d89a7" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714b61cb367d456f976b7b526a410367" is_pre_election: true
I20260619 16:35:38.070447 1669 raft_consensus.cc:2454] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 0.
I20260619 16:35:38.070470 1949 raft_consensus.cc:2454] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 0.
I20260619 16:35:38.070719 1741 leader_election.cc:302] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08dbc9d0624a4fbbb184c39f776dff2c, 720a24cf55884c4cb7ba2c83f901ca7d; no voters:
I20260619 16:35:38.070858 2038 raft_consensus.cc:2781] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:38.070914 2038 raft_consensus.cc:482] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:38.070989 2038 raft_consensus.cc:3037] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:38.071936 2038 raft_consensus.cc:504] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.072290 2038 leader_election.cc:288] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 election: Requested vote from peers 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585), 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749)
I20260619 16:35:38.072571 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c55a4d1de434cb77d20e4eb7d89a7" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720a24cf55884c4cb7ba2c83f901ca7d"
I20260619 16:35:38.072690 1669 raft_consensus.cc:3037] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:38.072854 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a5c55a4d1de434cb77d20e4eb7d89a7" candidate_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714b61cb367d456f976b7b526a410367"
I20260619 16:35:38.072954 1949 raft_consensus.cc:3037] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:38.073859 1669 raft_consensus.cc:2454] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 1.
I20260619 16:35:38.074052 1949 raft_consensus.cc:2454] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08dbc9d0624a4fbbb184c39f776dff2c in term 1.
I20260619 16:35:38.074149 1741 leader_election.cc:302] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08dbc9d0624a4fbbb184c39f776dff2c, 720a24cf55884c4cb7ba2c83f901ca7d; no voters:
I20260619 16:35:38.074388 2038 raft_consensus.cc:2781] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:38.074493 2038 raft_consensus.cc:686] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 LEADER]: Becoming Leader. State: Replica: 08dbc9d0624a4fbbb184c39f776dff2c, State: Running, Role: LEADER
I20260619 16:35:38.074620 2038 consensus_queue.cc:237] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.076287 1511 catalog_manager.cc:5495] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:35:38 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:38.112000 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.102994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33235 (local address 127.0.0.1:35060)
0619 16:35:38.103296 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:38.103309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:38.103400 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:38.103709 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:38.103712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:38.104079 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:38.104181 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.104187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.105189 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.105192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.105825 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.105833 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.106367 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.106369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.106438 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.107045 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:38.107056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:38.108660 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:38.110532 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:38.110539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:38.110544 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:38.110758 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:38.111047 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:38.111049 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:38.111050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:38.111142 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:38.111559 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:38.111564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:38.111661 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 16:35:38.111760 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:35:38.111869 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260619 16:35:38.112152 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.103049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.98.62:33235)
0619 16:35:38.103209 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.103214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.103383 (+ 169us) server_negotiation.cc:407] Connection header received
0619 16:35:38.103502 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.103505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.103550 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.103642 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:38.104316 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.105067 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.105944 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.106254 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.106360 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.108799 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:38.108803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:38.108806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:38.108830 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:38.110425 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:38.110848 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:38.110851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:38.110853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:38.110931 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:38.111241 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:38.111244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:38.111245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:38.111451 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:38.111554 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.111840 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.111957 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:35:38.116101 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:35:38.116828 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:38.122601 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3c8841f3024df3b21072f3657f4ce8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=817e5aba2ce14ea4919a901bc8499643]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:35:38.122601 1783 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3c8841f3024df3b21072f3657f4ce8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=817e5aba2ce14ea4919a901bc8499643]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:35:38.122988 1783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3c8841f3024df3b21072f3657f4ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:38.122988 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3c8841f3024df3b21072f3657f4ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:38.125830 2020 tablet_bootstrap.cc:492] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: Bootstrap starting.
I20260619 16:35:38.125978 2019 tablet_bootstrap.cc:492] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: Bootstrap starting.
I20260619 16:35:38.126322 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3c8841f3024df3b21072f3657f4ce8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=817e5aba2ce14ea4919a901bc8499643]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:35:38.126681 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3c8841f3024df3b21072f3657f4ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:38.126849 2020 tablet_bootstrap.cc:654] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:38.126907 2019 tablet_bootstrap.cc:654] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:38.128461 2020 tablet_bootstrap.cc:492] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: No bootstrap required, opened a new log
I20260619 16:35:38.128553 2020 ts_tablet_manager.cc:1397] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:35:38.128979 2020 raft_consensus.cc:348] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.129101 2020 raft_consensus.cc:374] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:38.129148 2020 raft_consensus.cc:729] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 720a24cf55884c4cb7ba2c83f901ca7d, State: Initialized, Role: FOLLOWER
I20260619 16:35:38.129302 2020 consensus_queue.cc:260] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.129618 2026 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.129753 2026 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.130319 2018 tablet_bootstrap.cc:492] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: Bootstrap starting.
I20260619 16:35:38.130731 2019 tablet_bootstrap.cc:492] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: No bootstrap required, opened a new log
I20260619 16:35:38.130833 2019 ts_tablet_manager.cc:1397] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:35:38.131275 2018 tablet_bootstrap.cc:654] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:38.131202 2019 raft_consensus.cc:348] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.131323 2019 raft_consensus.cc:374] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:38.131362 2019 raft_consensus.cc:729] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08dbc9d0624a4fbbb184c39f776dff2c, State: Initialized, Role: FOLLOWER
I20260619 16:35:38.131489 2019 consensus_queue.cc:260] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.131760 2019 ts_tablet_manager.cc:1428] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:38.131948 2038 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.132071 2038 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.132454 2020 ts_tablet_manager.cc:1428] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260619 16:35:38.133297 2018 tablet_bootstrap.cc:492] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: No bootstrap required, opened a new log
I20260619 16:35:38.133402 2018 ts_tablet_manager.cc:1397] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 16:35:38.133843 2018 raft_consensus.cc:348] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.133952 2018 raft_consensus.cc:374] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:38.133989 2018 raft_consensus.cc:729] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714b61cb367d456f976b7b526a410367, State: Initialized, Role: FOLLOWER
I20260619 16:35:38.134102 2018 consensus_queue.cc:260] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.134387 2018 ts_tablet_manager.cc:1428] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:35:38.134334 2025 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.134451 2025 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.146862 2025 raft_consensus.cc:482] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:38.146970 2025 raft_consensus.cc:504] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.147974 2025 leader_election.cc:288] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585), 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2:42289)
I20260619 16:35:38.152420 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.148146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43585 (local address 127.0.98.3:53989)
0619 16:35:38.148248 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:38.148263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:38.148345 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:38.148693 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:38.148697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:38.148716 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:38.149205 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.149215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.150224 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.150227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.151339 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.151349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.151973 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.151981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.152084 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.152121 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 16:35:38.152189 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 16:35:38.152265 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":22}
I20260619 16:35:38.152480 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.148277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53989 (local address 127.0.98.1:43585)
0619 16:35:38.148385 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.148390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.148404 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:35:38.148451 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.148454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.148497 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.148576 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:38.149382 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.150095 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.151466 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.151849 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.151969 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.152039 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.152282 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.152362 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:35:38.152565 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.148734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43109 (local address 127.0.98.2:42289)
0619 16:35:38.148853 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.148857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.148869 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:35:38.148918 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.148920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.148951 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.149044 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:38.149344 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.149853 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.151425 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.152029 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.152145 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.152206 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.152380 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.152455 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:35:38.152871 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3c8841f3024df3b21072f3657f4ce8" candidate_uuid: "714b61cb367d456f976b7b526a410367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" is_pre_election: true
I20260619 16:35:38.152949 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.148371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42289 (local address 127.0.98.3:43109)
0619 16:35:38.148667 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:38.148680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:38.148768 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:38.149048 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:38.149051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:38.149066 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:38.149183 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.149188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.149960 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.149966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.151298 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.151309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.152146 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.152152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.152249 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.152266 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:35:38.152724 (+ 458us) client_negotiation.cc:240] Negotiation successful
0619 16:35:38.152814 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":33,"thread_start_us":112,"threads_started":1}
I20260619 16:35:38.153012 1669 raft_consensus.cc:2454] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 714b61cb367d456f976b7b526a410367 in term 0.
I20260619 16:35:38.153810 1881 leader_election.cc:302] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 714b61cb367d456f976b7b526a410367, 720a24cf55884c4cb7ba2c83f901ca7d; no voters:
I20260619 16:35:38.153887 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3c8841f3024df3b21072f3657f4ce8" candidate_uuid: "714b61cb367d456f976b7b526a410367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" is_pre_election: true
I20260619 16:35:38.154081 1809 raft_consensus.cc:2454] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 714b61cb367d456f976b7b526a410367 in term 0.
I20260619 16:35:38.154328 2025 raft_consensus.cc:2781] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:38.154408 2025 raft_consensus.cc:482] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:38.154454 2025 raft_consensus.cc:3037] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:38.155391 2025 raft_consensus.cc:504] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.155757 2025 leader_election.cc:288] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [CANDIDATE]: Term 1 election: Requested vote from peers 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585), 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2:42289)
I20260619 16:35:38.156059 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3c8841f3024df3b21072f3657f4ce8" candidate_uuid: "714b61cb367d456f976b7b526a410367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "720a24cf55884c4cb7ba2c83f901ca7d"
I20260619 16:35:38.156093 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3c8841f3024df3b21072f3657f4ce8" candidate_uuid: "714b61cb367d456f976b7b526a410367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08dbc9d0624a4fbbb184c39f776dff2c"
I20260619 16:35:38.156178 1669 raft_consensus.cc:3037] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:38.156183 1809 raft_consensus.cc:3037] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:38.157227 1809 raft_consensus.cc:2454] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 714b61cb367d456f976b7b526a410367 in term 1.
I20260619 16:35:38.157227 1669 raft_consensus.cc:2454] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 714b61cb367d456f976b7b526a410367 in term 1.
I20260619 16:35:38.157541 1881 leader_election.cc:302] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08dbc9d0624a4fbbb184c39f776dff2c, 714b61cb367d456f976b7b526a410367; no voters:
I20260619 16:35:38.157707 2025 raft_consensus.cc:2781] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:38.157929 2025 raft_consensus.cc:686] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [term 1 LEADER]: Becoming Leader. State: Replica: 714b61cb367d456f976b7b526a410367, State: Running, Role: LEADER
I20260619 16:35:38.158169 2025 consensus_queue.cc:237] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } }
I20260619 16:35:38.159029 2025 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: TxnStatusTablet state changed. Reason: New leader 714b61cb367d456f976b7b526a410367. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.159116 2025 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:35:38.159379 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:35:38.159564 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:35:38.160481 1510 catalog_manager.cc:5495] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 reported cstate change: term changed from 0 to 1, leader changed from <none> to 714b61cb367d456f976b7b526a410367 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } health_report { overall_health: HEALTHY } } }
I20260619 16:35:38.177454 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.172654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33749 (local address 127.0.0.1:44844)
0619 16:35:38.172883 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:38.172897 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:38.173006 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:38.173401 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:38.173404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:38.173413 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:38.173540 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.173546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.174674 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.174678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.175641 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.175652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.176244 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.176248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.176345 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.176401 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:38.177129 (+ 728us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:38.177143 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:35:38.177213 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:35:38.177294 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 16:35:38.177588 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.172749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.98.3:33749)
0619 16:35:38.172861 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.172866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.172952 (+ 86us) server_negotiation.cc:407] Connection header received
0619 16:35:38.173172 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.173175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.173238 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.173367 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:38.173673 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.174552 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.175777 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.176116 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.176235 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.176553 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:38.177027 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:38.177140 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.177408 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.177462 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:35:38.180662 1809 raft_consensus.cc:1261] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Refusing update from remote peer 714b61cb367d456f976b7b526a410367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:38.181166 1669 raft_consensus.cc:1261] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Refusing update from remote peer 714b61cb367d456f976b7b526a410367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:38.181690 2025 consensus_queue.cc:1035] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:38.182075 2058 consensus_queue.cc:1035] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:38.183387 2026 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: TxnStatusTablet state changed. Reason: New leader 714b61cb367d456f976b7b526a410367. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.183495 2026 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.183565 2038 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: TxnStatusTablet state changed. Reason: New leader 714b61cb367d456f976b7b526a410367. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.183672 2038 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.189702 2025 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.189810 2025 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:35:38.189924 2025 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.189987 2025 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 714b61cb367d456f976b7b526a410367: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:35:38.190331 2038 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.190428 2038 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 08dbc9d0624a4fbbb184c39f776dff2c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.191376 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298608898782552064 to 7298608898697699328. Current Snapshot: MvccSnapshot[applied={T|T < 7298608898782552064}]
I20260619 16:35:38.191504 2026 tablet_replica.cc:439] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "714b61cb367d456f976b7b526a410367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 } } peers { permanent_uuid: "08dbc9d0624a4fbbb184c39f776dff2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42289 } } peers { permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 } } }
I20260619 16:35:38.191617 2026 tablet_replica.cc:442] T 8e3c8841f3024df3b21072f3657f4ce8 P 720a24cf55884c4cb7ba2c83f901ca7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:38.208487 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.201955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47955 (local address 127.0.98.3:33749)
0619 16:35:38.202133 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.202137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.205602 (+ 3465us) server_negotiation.cc:407] Connection header received
0619 16:35:38.205753 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.205756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.205800 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.205884 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:38.206111 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.206592 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.207452 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.207704 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.207814 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.208084 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:38.208158 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:38.208232 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.208324 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.208366 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:35:38.208518 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.201797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33749 (local address 127.0.98.2:47955)
0619 16:35:38.205515 (+ 3718us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:38.205528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:38.205646 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:38.205893 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:38.205896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:38.205903 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:38.206003 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.206008 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.206704 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.206709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.207331 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.207340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.207813 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.207817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.207900 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.207950 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:38.208246 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:38.208253 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:35:38.208314 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 16:35:38.208390 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3633,"thread_start_us":155,"threads_started":1}
I20260619 16:35:38.214636 1949 raft_consensus.cc:1261] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Refusing update from remote peer 08dbc9d0624a4fbbb184c39f776dff2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:38.214756 1669 raft_consensus.cc:1261] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Refusing update from remote peer 08dbc9d0624a4fbbb184c39f776dff2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:38.215018 2038 consensus_queue.cc:1035] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "714b61cb367d456f976b7b526a410367" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:38.215147 2028 consensus_queue.cc:1035] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "720a24cf55884c4cb7ba2c83f901ca7d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:38.234408 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.230608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42289 (local address 127.0.98.3:33037)
0619 16:35:38.230847 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:38.230863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:38.230963 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:38.231235 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:38.231238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:38.231246 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:38.231358 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.231363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.232223 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.232227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.233201 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:38.233212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.233718 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.233721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:38.233817 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.233865 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:38.234153 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:38.234163 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 16:35:38.234227 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:35:38.234300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 16:35:38.234409 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.230680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33037 (local address 127.0.98.2:42289)
0619 16:35:38.230786 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.230791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.230933 (+ 142us) server_negotiation.cc:407] Connection header received
0619 16:35:38.231088 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.231091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.231137 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.231214 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:38.231471 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.232119 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.233305 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.233609 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.233690 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.233993 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:38.234071 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:38.234149 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.234236 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.234284 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:35:38.252856 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:38.252983 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:38.254851 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3911884113f24b4397be2da7baddf89c: Sending DeleteTablet for 3 replicas of tablet 3a5c55a4d1de434cb77d20e4eb7d89a7
I20260619 16:35:38.255308 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44124
I20260619 16:35:38.255451 1783 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58690
I20260619 16:35:38.255503 2041 tablet_replica.cc:330] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: stopping tablet replica
I20260619 16:35:38.255611 2041 raft_consensus.cc:2229] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:38.255713 2041 raft_consensus.cc:2258] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:38.255753 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44834
I20260619 16:35:38.255860 2043 tablet_replica.cc:330] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: stopping tablet replica
I20260619 16:35:38.255939 2042 tablet_replica.cc:330] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: stopping tablet replica
I20260619 16:35:38.255957 2043 raft_consensus.cc:2229] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:38.256038 2042 raft_consensus.cc:2229] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:38.256112 2043 raft_consensus.cc:2258] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:38.256160 2042 raft_consensus.cc:2258] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:38.256997 2042 ts_tablet_manager.cc:1905] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:38.257149 2041 ts_tablet_manager.cc:1905] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:38.257695 2043 ts_tablet_manager.cc:1905] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:38.259891 2042 ts_tablet_manager.cc:1918] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:38.259958 2042 log.cc:1196] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/3a5c55a4d1de434cb77d20e4eb7d89a7
I20260619 16:35:38.260150 2043 ts_tablet_manager.cc:1918] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:38.260212 2043 log.cc:1196] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/3a5c55a4d1de434cb77d20e4eb7d89a7
I20260619 16:35:38.260226 2042 ts_tablet_manager.cc:1939] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 714b61cb367d456f976b7b526a410367: Deleting consensus metadata
I20260619 16:35:38.260453 2043 ts_tablet_manager.cc:1939] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 08dbc9d0624a4fbbb184c39f776dff2c: Deleting consensus metadata
I20260619 16:35:38.260993 1498 catalog_manager.cc:4841] TS 714b61cb367d456f976b7b526a410367 (127.0.98.3:33749): tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 (table test-table [id=732f6dfb56a74192a461954eb77c839a]) successfully deleted
I20260619 16:35:38.263384 2041 ts_tablet_manager.cc:1918] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:38.263463 2041 log.cc:1196] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/3a5c55a4d1de434cb77d20e4eb7d89a7
I20260619 16:35:38.263700 2041 ts_tablet_manager.cc:1939] T 3a5c55a4d1de434cb77d20e4eb7d89a7 P 720a24cf55884c4cb7ba2c83f901ca7d: Deleting consensus metadata
I20260619 16:35:38.263943 1497 catalog_manager.cc:4841] TS 08dbc9d0624a4fbbb184c39f776dff2c (127.0.98.2:42289): tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 (table test-table [id=732f6dfb56a74192a461954eb77c839a]) successfully deleted
Jun 19 16:35:38 dist-test-slave-0f28 krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886937, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:38.264487 1497 catalog_manager.cc:4841] TS 720a24cf55884c4cb7ba2c83f901ca7d (127.0.98.1:43585): tablet 3a5c55a4d1de434cb77d20e4eb7d89a7 (table test-table [id=732f6dfb56a74192a461954eb77c839a]) successfully deleted
I20260619 16:35:38.268407 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.257038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44134 (local address 127.0.98.1:43585)
0619 16:35:38.257160 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.257164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.257698 (+ 534us) server_negotiation.cc:407] Connection header received
0619 16:35:38.257865 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.257868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.257909 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.257956 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:38.258905 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.259679 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.260874 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.261208 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.261288 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.264894 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:38.264903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:38.264919 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:38.264943 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:38.266810 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:38.267185 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:38.267192 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:38.267194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:38.267294 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:38.267530 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:38.267533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:38.267536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:38.267805 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:38.268005 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.268131 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.268240 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 16:35:38.268874 1709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44134
I20260619 16:35:38.278676 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:38.270473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.0.98.62:33235)
0619 16:35:38.270581 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:38.270585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:38.270703 (+ 118us) server_negotiation.cc:407] Connection header received
0619 16:35:38.270791 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:38.270794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:38.270838 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:38.270908 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:38.271442 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.272219 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.273672 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:38.274075 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:38.274184 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:38.275420 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:38.275424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:38.275427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:38.275455 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:38.277086 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:38.277564 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:38.277568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:38.277570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:38.277655 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:38.277899 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:38.277902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:38.277903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:38.278106 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:38.278235 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:38.278417 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 16:35:38.278526 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:35:38.279069 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35064
I20260619 16:35:38.280325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1584
I20260619 16:35:38.288098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1724
I20260619 16:35:38.294757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1864
I20260619 16:35:38.301235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 1480
2026-06-19T16:35:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4066 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:38 dist-test-slave-0f28 krb5kdc[2080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:38 dist-test-slave-0f28 krb5kdc[2080](info): set up 2 sockets
Jun 19 16:35:38 dist-test-slave-0f28 krb5kdc[2080](info): commencing operation
krb5kdc: starting...
W20260619 16:35:39.353030 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:39 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:39.477269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:39.581091 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:39.581354 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:39.581393 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:39.585250 2093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:39.585299 2093 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:35:39.585367 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:39.585407 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:39.585423 2093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:39.585444 2093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:39.590404 2093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33347
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:39.591269 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:39.592285 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:39.600440 2098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:39.600451 2101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:39.600593 2093 server_base.cc:1034] running on GCE node
W20260619 16:35:39.600791 2099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:39.601382 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:39.602391 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:39.603556 2093 hybrid_clock.cc:648] HybridClock initialized: now 1781886939603537 us; error 31 us; skew 500 ppm
Jun 19 16:35:39 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:39.606175 2093 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:39.606984 2093 webserver.cc:460] Webserver started at http://127.0.98.62:40903/ using document root <none> and password file <none>
I20260619 16:35:39.607507 2093 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:39.607597 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:39.607834 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:39.609795 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "47fff942ff2b4eb0897443bdf4def88b"
format_stamp: "Formatted at 2026-06-19 16:35:39 on dist-test-slave-0f28"
server_key: "54f8913b9f2e92ea41d9b6cf63b055e3"
server_key_iv: "1081dd689ee2bdf95c34aa329b64a8a0"
server_key_version: "encryptionkey@0"
I20260619 16:35:39.610343 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "47fff942ff2b4eb0897443bdf4def88b"
format_stamp: "Formatted at 2026-06-19 16:35:39 on dist-test-slave-0f28"
server_key: "54f8913b9f2e92ea41d9b6cf63b055e3"
server_key_iv: "1081dd689ee2bdf95c34aa329b64a8a0"
server_key_version: "encryptionkey@0"
I20260619 16:35:39.613798 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:35:39.616016 2108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:39.617158 2093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:39.617297 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "47fff942ff2b4eb0897443bdf4def88b"
format_stamp: "Formatted at 2026-06-19 16:35:39 on dist-test-slave-0f28"
server_key: "54f8913b9f2e92ea41d9b6cf63b055e3"
server_key_iv: "1081dd689ee2bdf95c34aa329b64a8a0"
server_key_version: "encryptionkey@0"
I20260619 16:35:39.617457 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:39.639827 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:39.642632 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:39.642836 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:39.650123 2093 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33347
I20260619 16:35:39.650147 2170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33347 every 8 connection(s)
I20260619 16:35:39.650979 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:39.652640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 2093
I20260619 16:35:39.652732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:39.652987 392 external_mini_cluster.cc:1428] Setting key 7ed2bb11b504b8c06bf39ce5499a7fc9
I20260619 16:35:39.653775 2171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:39.660140 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Bootstrap starting.
Jun 19 16:35:39 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:39.662528 2171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:39.663218 2171 log.cc:824] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:39.664893 2171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: No bootstrap required, opened a new log
I20260619 16:35:39.667600 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:39.654308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.0.98.62:33347)
0619 16:35:39.654694 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:39.654712 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:39.654750 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:35:39.655587 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:39.655608 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:39.655993 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:39.656340 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:39.656967 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.658449 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.659279 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.659926 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.660085 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:39.662723 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:39.662727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:39.662739 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:39.662765 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:39.664961 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:39.665433 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:39.665439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:39.665447 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:39.665557 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:39.665850 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:39.665853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:39.665855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:39.666206 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:39.666407 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:39.666673 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 16:35:39.666939 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20260619 16:35:39.667810 2171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47fff942ff2b4eb0897443bdf4def88b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33347 } }
I20260619 16:35:39.668056 2171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:39.668149 2171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fff942ff2b4eb0897443bdf4def88b, State: Initialized, Role: FOLLOWER
I20260619 16:35:39.668581 2171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47fff942ff2b4eb0897443bdf4def88b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33347 } }
I20260619 16:35:39.668727 2171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:39.668793 2171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:39.668893 2171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:39.669842 2171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47fff942ff2b4eb0897443bdf4def88b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33347 } }
I20260619 16:35:39.670182 2171 leader_election.cc:302] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47fff942ff2b4eb0897443bdf4def88b; no voters:
I20260619 16:35:39.670454 2171 leader_election.cc:288] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:39.670567 2176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:39.670806 2176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [term 1 LEADER]: Becoming Leader. State: Replica: 47fff942ff2b4eb0897443bdf4def88b, State: Running, Role: LEADER
I20260619 16:35:39.671093 2176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47fff942ff2b4eb0897443bdf4def88b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33347 } }
I20260619 16:35:39.671587 2171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:39.672756 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47fff942ff2b4eb0897443bdf4def88b. Latest consensus state: current_term: 1 leader_uuid: "47fff942ff2b4eb0897443bdf4def88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47fff942ff2b4eb0897443bdf4def88b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33347 } } }
I20260619 16:35:39.672905 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:39.672756 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47fff942ff2b4eb0897443bdf4def88b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47fff942ff2b4eb0897443bdf4def88b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33347 } } }
I20260619 16:35:39.673039 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:39.673748 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:39.676442 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:39.680930 2186 catalog_manager.cc:1347] Generated new cluster ID: c67671e0145746b4a789e03d043028a7
I20260619 16:35:39.680989 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:39.690917 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:39.691736 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:39.697422 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Generated new TSK 0
I20260619 16:35:39.698143 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:39.724743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33347
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:39.831213 2197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:39.831445 2197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:39.831489 2197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:39.835453 2197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:39.835501 2197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:39.835538 2197 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:35:39.835552 2197 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:35:39.835583 2197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:39.840566 2197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:39.841516 2197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:39.842469 2197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:35:39.851830 2197 server_base.cc:1034] running on GCE node
W20260619 16:35:39.851864 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:39.851864 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:39.851938 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:39.852371 2197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:39.852905 2197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:39.854112 2197 hybrid_clock.cc:648] HybridClock initialized: now 1781886939854085 us; error 39 us; skew 500 ppm
Jun 19 16:35:39 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:39.856777 2197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:39.857694 2197 webserver.cc:460] Webserver started at http://127.0.98.1:35779/ using document root <none> and password file <none>
I20260619 16:35:39.858256 2197 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:39.858372 2197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:39.858644 2197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:39.860574 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "efb7c6f830194368b1b6a11de3df809d"
format_stamp: "Formatted at 2026-06-19 16:35:39 on dist-test-slave-0f28"
server_key: "bd58b3b2226e3886dd406d1ea165d1a0"
server_key_iv: "ad6a505f3bd00b19e7841104f4816179"
server_key_version: "encryptionkey@0"
I20260619 16:35:39.861115 2197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "efb7c6f830194368b1b6a11de3df809d"
format_stamp: "Formatted at 2026-06-19 16:35:39 on dist-test-slave-0f28"
server_key: "bd58b3b2226e3886dd406d1ea165d1a0"
server_key_iv: "ad6a505f3bd00b19e7841104f4816179"
server_key_version: "encryptionkey@0"
I20260619 16:35:39.864676 2197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:35:39.867450 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:39.868896 2197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:35:39.869040 2197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "efb7c6f830194368b1b6a11de3df809d"
format_stamp: "Formatted at 2026-06-19 16:35:39 on dist-test-slave-0f28"
server_key: "bd58b3b2226e3886dd406d1ea165d1a0"
server_key_iv: "ad6a505f3bd00b19e7841104f4816179"
server_key_version: "encryptionkey@0"
I20260619 16:35:39.869174 2197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:39.883548 2197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:39.886284 2197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:39.886476 2197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:39.887856 2197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:39.887928 2197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:39.887993 2197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:39.888032 2197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:35:39 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:39.899168 2197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45919
I20260619 16:35:39.899354 2327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45919 every 8 connection(s)
I20260619 16:35:39.900070 2197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:39.900264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 2197
I20260619 16:35:39.900388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:39.900691 392 external_mini_cluster.cc:1428] Setting key 97729998084412acf76a47348b4ffb8a
I20260619 16:35:39.902824 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:39.888476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51695 (local address 127.0.98.62:33347)
0619 16:35:39.888617 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:39.888622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:39.889495 (+ 873us) server_negotiation.cc:407] Connection header received
0619 16:35:39.890461 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:39.890464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:39.890521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:39.890603 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:39.892058 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.892970 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.895328 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.895766 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.895924 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:39.898720 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:39.898723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:39.898726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:39.898751 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:39.900483 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:39.901082 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:39.901085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:39.901087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:39.901170 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:39.901614 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:39.901616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:39.901617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:39.901785 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:39.901859 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:39.902466 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 16:35:39.902596 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:35:39.903420 2221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:39.888890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.98.1:51695)
0619 16:35:39.889307 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:39.889380 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:39.890261 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:39.890869 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:39.890877 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:39.891338 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:39.891886 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:39.891903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.893133 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.893136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:39.895145 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:39.895158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.895971 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.895975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:39.896168 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:39.896786 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:39.896809 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:39.898556 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:39.900618 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:39.900625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:39.900639 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:39.900965 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:39.901282 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:39.901285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:39.901287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:39.901504 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:39.901893 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:39.901899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:39.902185 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 16:35:39.902365 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 16:35:39.902597 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":122,"threads_started":1}
I20260619 16:35:39.904851 2328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33347
I20260619 16:35:39.904999 2328 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:39.905587 2328 heartbeater.cc:511] Master 127.0.98.62:33347 requested a full tablet report, sending...
I20260619 16:35:39.907174 2124 ts_manager.cc:194] Registered new tserver with Master: efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919)
I20260619 16:35:39.908829 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51695
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:35:39.918614 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:39.909684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52771 (local address 127.0.98.62:33347)
0619 16:35:39.909818 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:39.909822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:39.911574 (+ 1752us) server_negotiation.cc:407] Connection header received
0619 16:35:39.911640 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:39.911642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:39.911679 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:39.911741 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:39.912073 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.912598 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.913366 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.913639 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.913729 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:39.914843 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:39.914850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:39.914852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:39.914881 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:39.916835 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:39.917407 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:39.917410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:39.917411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:39.917485 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:39.917826 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:39.917830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:39.917833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:39.918031 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:39.918109 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:39.918414 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:35:39.918506 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:35:39.918785 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:39.909645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.98.1:52771)
0619 16:35:39.911493 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:39.911505 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:39.911582 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:39.911750 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:39.911751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:39.911904 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:39.911986 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:39.911990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.912687 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.912689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:39.913227 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:39.913235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:39.913734 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:39.913736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:39.913803 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:39.914296 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:39.914307 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:39.914669 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:39.916952 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:39.916958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:39.916963 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:39.917238 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:39.917584 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:39.917590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:39.917595 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:39.917697 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:39.918131 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:39.918135 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:39.918252 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:35:39.918369 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:35:39.918532 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1774,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:39.931949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33347
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:40.036602 2337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:40.036890 2337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:40.036972 2337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:40.040853 2337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:40.040939 2337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:40.041013 2337 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:35:40.041059 2337 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:35:40.041119 2337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:40.046661 2337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:40.047645 2337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:40.048604 2337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:35:40.057780 2337 server_base.cc:1034] running on GCE node
W20260619 16:35:40.057667 2345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:40.057682 2343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:40.057658 2342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:40.058310 2337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:40.058930 2337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:40.060118 2337 hybrid_clock.cc:648] HybridClock initialized: now 1781886940060106 us; error 26 us; skew 500 ppm
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:40.062891 2337 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:40.063694 2337 webserver.cc:460] Webserver started at http://127.0.98.2:37071/ using document root <none> and password file <none>
I20260619 16:35:40.064244 2337 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:40.064335 2337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:40.064561 2337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:40.066617 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "08a01c6775d541e08c342dc2a7f02c00"
format_stamp: "Formatted at 2026-06-19 16:35:40 on dist-test-slave-0f28"
server_key: "a9a3b2e8d600fa2571db9f4fb6dc7099"
server_key_iv: "88ee048dfdaf3dfa47cd779dda4aeedb"
server_key_version: "encryptionkey@0"
I20260619 16:35:40.067162 2337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "08a01c6775d541e08c342dc2a7f02c00"
format_stamp: "Formatted at 2026-06-19 16:35:40 on dist-test-slave-0f28"
server_key: "a9a3b2e8d600fa2571db9f4fb6dc7099"
server_key_iv: "88ee048dfdaf3dfa47cd779dda4aeedb"
server_key_version: "encryptionkey@0"
I20260619 16:35:40.070657 2337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:40.073192 2352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:40.074401 2337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:40.074522 2337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "08a01c6775d541e08c342dc2a7f02c00"
format_stamp: "Formatted at 2026-06-19 16:35:40 on dist-test-slave-0f28"
server_key: "a9a3b2e8d600fa2571db9f4fb6dc7099"
server_key_iv: "88ee048dfdaf3dfa47cd779dda4aeedb"
server_key_version: "encryptionkey@0"
I20260619 16:35:40.074636 2337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:40.088608 2337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:40.091336 2337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:40.091531 2337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:40.092866 2337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:40.092936 2337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:40.093001 2337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:40.093050 2337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:40.103842 2337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40015
I20260619 16:35:40.103922 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40015 every 8 connection(s)
I20260619 16:35:40.104686 2337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:40.107197 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.093804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34383 (local address 127.0.98.62:33347)
0619 16:35:40.093931 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.093935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.094890 (+ 955us) server_negotiation.cc:407] Connection header received
0619 16:35:40.096116 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.096119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.096186 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.096301 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:40.097884 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.098491 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.100373 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.100736 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.100849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.103737 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:40.103740 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:40.103749 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:40.103771 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:40.105579 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.106002 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.106004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.106005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.106077 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.106360 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.106362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.106362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.106520 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:40.106593 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.106987 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.107078 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:35:40.107234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 2337
I20260619 16:35:40.107331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:40.107597 392 external_mini_cluster.cc:1428] Setting key 838998c2fc2ad00f5bf1b5659cf65ab3
I20260619 16:35:40.108163 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.094292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.98.2:34383)
0619 16:35:40.094742 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.094785 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.095868 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.096613 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.096626 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.097043 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:40.097733 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.097749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.098649 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.098656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.100233 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.100248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.100897 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.100901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.101119 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.101700 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:40.101722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:40.103549 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:40.105658 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.105663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.105677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.105916 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.106157 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.106159 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.106161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.106283 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.106599 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:40.106603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:40.106798 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.106938 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.107284 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260619 16:35:40.109304 2468 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33347
I20260619 16:35:40.109462 2468 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:40.109925 2468 heartbeater.cc:511] Master 127.0.98.62:33347 requested a full tablet report, sending...
I20260619 16:35:40.110939 2123 ts_manager.cc:194] Registered new tserver with Master: 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.111737 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34383
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:40.118633 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.111592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51869 (local address 127.0.98.62:33347)
0619 16:35:40.112129 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.112132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.112143 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:35:40.112172 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.112175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.112209 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.112317 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:40.112699 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.113200 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.114100 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.114317 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.114422 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.115431 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:40.115434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:40.115437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:40.115458 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:40.116935 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.117427 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.117431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.117432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.117512 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.117841 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.117847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.117849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.118041 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:40.118146 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.118391 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.118489 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468}
I20260619 16:35:40.118645 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.111548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.98.2:51869)
0619 16:35:40.111779 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.111791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.111855 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.112346 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.112348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.112498 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:40.112586 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.112591 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.113337 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.113343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.113990 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.113998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.114424 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.114427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.114520 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.115001 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:40.115011 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:40.115305 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:40.117067 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.117070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.117072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.117280 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.117651 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.117656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.117658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.117764 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.118135 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:40.118138 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:40.118230 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.118332 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.118460 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:40.136561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33347
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:40.245354 2477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:40.245622 2477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:40.245697 2477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:40.249754 2477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:40.249822 2477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:40.249903 2477 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 16:35:40.249953 2477 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:35:40.250015 2477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:40.255331 2477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:40.256266 2477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:40.257246 2477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:40.266532 2483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:40.266508 2482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:40.266659 2477 server_base.cc:1034] running on GCE node
W20260619 16:35:40.266523 2485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:40.267226 2477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:40.267786 2477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:40.268939 2477 hybrid_clock.cc:648] HybridClock initialized: now 1781886940268909 us; error 42 us; skew 500 ppm
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:40.271607 2477 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:40.272394 2477 webserver.cc:460] Webserver started at http://127.0.98.3:35571/ using document root <none> and password file <none>
I20260619 16:35:40.272987 2477 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:40.273061 2477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:40.273216 2477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:40.275167 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "eac8bbe947a3457ba38eb19fdc57de96"
format_stamp: "Formatted at 2026-06-19 16:35:40 on dist-test-slave-0f28"
server_key: "0e23329119740520983faeffe9a64486"
server_key_iv: "7b057f8cb589c1fd7616d505c28c0f9c"
server_key_version: "encryptionkey@0"
I20260619 16:35:40.275655 2477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "eac8bbe947a3457ba38eb19fdc57de96"
format_stamp: "Formatted at 2026-06-19 16:35:40 on dist-test-slave-0f28"
server_key: "0e23329119740520983faeffe9a64486"
server_key_iv: "7b057f8cb589c1fd7616d505c28c0f9c"
server_key_version: "encryptionkey@0"
I20260619 16:35:40.279184 2477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:40.281482 2492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:40.282737 2477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:40.282859 2477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "eac8bbe947a3457ba38eb19fdc57de96"
format_stamp: "Formatted at 2026-06-19 16:35:40 on dist-test-slave-0f28"
server_key: "0e23329119740520983faeffe9a64486"
server_key_iv: "7b057f8cb589c1fd7616d505c28c0f9c"
server_key_version: "encryptionkey@0"
I20260619 16:35:40.282960 2477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:40.325631 2477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:40.328884 2477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:40.329088 2477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:40.330494 2477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:40.330547 2477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:40.330587 2477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:40.330641 2477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:40.340807 2477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36195
I20260619 16:35:40.340880 2607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36195 every 8 connection(s)
I20260619 16:35:40.341776 2477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:40.342434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 2477
I20260619 16:35:40.342557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:40.342830 392 external_mini_cluster.cc:1428] Setting key 240918bb335e2f0ab21584d5c38c6eac
I20260619 16:35:40.344899 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.331430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57001 (local address 127.0.98.62:33347)
0619 16:35:40.331578 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.331583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.332397 (+ 814us) server_negotiation.cc:407] Connection header received
0619 16:35:40.333596 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.333599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.333663 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.333767 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:40.335352 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.336247 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.337799 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.338033 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.338144 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.340787 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:40.340793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:40.340797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:40.340830 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:40.342447 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.343408 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.343411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.343412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.343535 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.343867 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.343871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.343872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.344055 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:40.344156 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.344624 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.344738 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:35:40.345681 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.331816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.98.3:57001)
0619 16:35:40.332237 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.332284 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.333353 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.333983 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.333990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.334470 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:40.335182 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.335200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.336369 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.336371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.337668 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.337676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.338178 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.338182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.338360 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.338948 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:40.338967 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:40.340563 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:40.342982 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.342989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.343003 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.343292 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.343635 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.343637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.343638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.343763 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.344166 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:40.344175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:40.344402 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.344581 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.344879 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":117,"threads_started":1}
I20260619 16:35:40.346881 2608 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33347
I20260619 16:35:40.347045 2608 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:40.347615 2608 heartbeater.cc:511] Master 127.0.98.62:33347 requested a full tablet report, sending...
I20260619 16:35:40.348546 2124 ts_manager.cc:194] Registered new tserver with Master: eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195)
I20260619 16:35:40.349287 2124 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57001
I20260619 16:35:40.355593 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.348851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.98.3:56089)
0619 16:35:40.349094 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.349115 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.349190 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.349442 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.349445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.349716 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:40.349848 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.349855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.350561 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.350565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.351311 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.351319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.351712 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.351714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.351777 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.352199 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:40.352210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:40.352489 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:40.354294 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.354296 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.354298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.354468 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.354733 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.354735 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.354737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.354805 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.355103 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:40.355104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:40.355172 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.355293 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.355417 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
I20260619 16:35:40.355597 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.348926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56089 (local address 127.0.98.62:33347)
0619 16:35:40.349021 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.349025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.349179 (+ 154us) server_negotiation.cc:407] Connection header received
0619 16:35:40.349251 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.349253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.349293 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.349397 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:40.349956 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.350434 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.351407 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.351600 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.351671 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.352650 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:40.352653 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:40.352655 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:40.352675 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:40.354204 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.354572 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.354574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.354575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.354645 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.354878 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.354881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.354882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.355027 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:40.355098 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.355337 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.355429 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:35:40.357005 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:40.382464 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.374205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.0.98.62:33347)
0619 16:35:40.374330 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.374334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.374345 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:35:40.374383 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.374385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.374422 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.374514 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:40.375194 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.375703 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.376559 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.376780 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.376891 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.379420 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:40.379423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:40.379425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:40.379450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:40.380855 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.381192 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.381194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.381196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.381267 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.381759 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.381761 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.381762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.381928 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:40.382029 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.382199 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.382286 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:35:40.385255 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:40.388492 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:40.402530 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.397181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40015 (local address 127.0.0.1:60848)
0619 16:35:40.398194 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.398208 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.398299 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.398776 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.398779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.398796 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.398884 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.398890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.400062 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.400067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.401049 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.401060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.401994 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.401997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.402111 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.402151 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.402258 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.402374 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260619 16:35:40.402576 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.396606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45919 (local address 127.0.0.1:39566)
0619 16:35:40.397282 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.397297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.397388 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.398037 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.398044 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.398066 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.398194 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.398203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.399670 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.399673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.401002 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.401011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.402235 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.402253 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.402330 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.402349 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.402419 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.402473 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":98,"thread_start_us":97,"threads_started":1}
I20260619 16:35:40.402808 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.396501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36195 (local address 127.0.0.1:42310)
0619 16:35:40.397043 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.397090 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.397223 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.398199 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.398201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.398228 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.398348 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.398355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.399545 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.399551 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.401172 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.401179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.402370 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.402373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.402483 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.402504 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.402573 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.402645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":29,"thread_start_us":75,"threads_started":1}
I20260619 16:35:40.402920 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.396985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.0.98.1:45919)
0619 16:35:40.397377 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.397391 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.397415 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:35:40.397494 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.397499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.397710 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.397860 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.398341 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.399562 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.401128 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.401953 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.402059 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.402223 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.402469 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.402573 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":79,"threads_started":1}
I20260619 16:35:40.403226 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.396602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.0.98.3:36195)
0619 16:35:40.397534 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.397542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.397560 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:35:40.397617 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.397621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.398051 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.398488 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.398600 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.399439 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.401288 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.402091 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.402178 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.402319 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.402999 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.403098 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":94,"threads_started":1}
I20260619 16:35:40.403318 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.397548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60848 (local address 127.0.98.2:40015)
0619 16:35:40.397887 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.397905 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.398388 (+ 483us) server_negotiation.cc:407] Connection header received
0619 16:35:40.398462 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.398468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.398633 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.398787 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.399006 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.399953 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.401183 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.401871 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.401997 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.402159 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.403047 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.403179 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260619 16:35:40.405885 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 63d8082adb80497b81fbbb27aab6775d (DEFAULT_TABLE table=test-table [id=82bbed1fc62d4d5eb0f505d71e3ca8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:40.405866 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 63d8082adb80497b81fbbb27aab6775d (DEFAULT_TABLE table=test-table [id=82bbed1fc62d4d5eb0f505d71e3ca8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:40.406807 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d8082adb80497b81fbbb27aab6775d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.406840 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d8082adb80497b81fbbb27aab6775d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.407027 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 63d8082adb80497b81fbbb27aab6775d (DEFAULT_TABLE table=test-table [id=82bbed1fc62d4d5eb0f505d71e3ca8ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:40.407902 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d8082adb80497b81fbbb27aab6775d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.413362 2631 tablet_bootstrap.cc:492] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Bootstrap starting.
I20260619 16:35:40.414414 2632 tablet_bootstrap.cc:492] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Bootstrap starting.
I20260619 16:35:40.414809 2633 tablet_bootstrap.cc:492] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Bootstrap starting.
I20260619 16:35:40.415795 2631 tablet_bootstrap.cc:654] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.416640 2632 tablet_bootstrap.cc:654] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.416623 2631 log.cc:824] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:40.416785 2633 tablet_bootstrap.cc:654] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.417492 2632 log.cc:824] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:40.417539 2633 log.cc:824] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:40.418854 2631 tablet_bootstrap.cc:492] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: No bootstrap required, opened a new log
I20260619 16:35:40.419041 2631 ts_tablet_manager.cc:1397] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:35:40.419294 2632 tablet_bootstrap.cc:492] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: No bootstrap required, opened a new log
I20260619 16:35:40.419369 2633 tablet_bootstrap.cc:492] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: No bootstrap required, opened a new log
I20260619 16:35:40.419481 2632 ts_tablet_manager.cc:1397] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:35:40.419538 2633 ts_tablet_manager.cc:1397] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 16:35:40.422387 2632 raft_consensus.cc:348] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.422605 2632 raft_consensus.cc:374] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.422655 2632 raft_consensus.cc:729] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.422932 2631 raft_consensus.cc:348] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.423174 2631 raft_consensus.cc:374] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.423228 2631 raft_consensus.cc:729] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a01c6775d541e08c342dc2a7f02c00, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.423136 2632 consensus_queue.cc:260] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.423162 2633 raft_consensus.cc:348] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.423398 2633 raft_consensus.cc:374] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.423452 2633 raft_consensus.cc:729] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.423681 2631 consensus_queue.cc:260] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.423784 2633 consensus_queue.cc:260] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.424100 2632 ts_tablet_manager.cc:1428] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:35:40.424419 2328 heartbeater.cc:503] Master 127.0.98.62:33347 was elected leader, sending a full tablet report...
I20260619 16:35:40.424412 2468 heartbeater.cc:503] Master 127.0.98.62:33347 was elected leader, sending a full tablet report...
I20260619 16:35:40.424599 2633 ts_tablet_manager.cc:1428] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 16:35:40.424994 2608 heartbeater.cc:503] Master 127.0.98.62:33347 was elected leader, sending a full tablet report...
I20260619 16:35:40.426025 2631 ts_tablet_manager.cc:1428] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:35:40.436334 2637 raft_consensus.cc:482] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:40.436506 2637 raft_consensus.cc:504] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.437502 2637 leader_election.cc:288] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195), 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.442625 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.437730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42501 (local address 127.0.98.3:36195)
0619 16:35:40.437897 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.437902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.438045 (+ 143us) server_negotiation.cc:407] Connection header received
0619 16:35:40.438194 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.438197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.438255 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.438335 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.438779 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.439566 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.441426 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.442034 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.442142 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.442200 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.442437 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.442506 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 16:35:40.442625 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.437864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47425 (local address 127.0.98.2:40015)
0619 16:35:40.437976 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.437986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.438163 (+ 177us) server_negotiation.cc:407] Connection header received
0619 16:35:40.438212 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.438215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.438255 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.438335 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.438779 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.439565 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.441434 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.442033 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.442142 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.442200 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.442437 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.442506 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:35:40.442721 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.437652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36195 (local address 127.0.98.1:42501)
0619 16:35:40.437954 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.437969 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.438074 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.438360 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.438363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.438403 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.438671 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.438678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.439682 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.439686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.441294 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.441305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.442153 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.442157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.442307 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.442333 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.442412 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.442532 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":307,"spinlock_wait_cycles":10880,"thread_start_us":92,"threads_started":1}
I20260619 16:35:40.442903 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.437733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40015 (local address 127.0.98.1:47425)
0619 16:35:40.438053 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.438065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.438150 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.438360 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.438363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.438401 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.438671 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.438678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.439682 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.439686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.441294 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.441306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.442153 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.442157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.442311 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.442335 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.442412 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.442536 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":258,"spinlock_wait_cycles":29056,"thread_start_us":102,"threads_started":1}
I20260619 16:35:40.443348 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d8082adb80497b81fbbb27aab6775d" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8bbe947a3457ba38eb19fdc57de96" is_pre_election: true
I20260619 16:35:40.443465 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d8082adb80497b81fbbb27aab6775d" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00" is_pre_election: true
I20260619 16:35:40.443655 2562 raft_consensus.cc:2454] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 0.
I20260619 16:35:40.443763 2422 raft_consensus.cc:2454] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 0.
I20260619 16:35:40.444147 2213 leader_election.cc:302] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eac8bbe947a3457ba38eb19fdc57de96, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:40.444389 2637 raft_consensus.cc:2781] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:40.444468 2637 raft_consensus.cc:482] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:40.444510 2637 raft_consensus.cc:3037] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.445526 2637 raft_consensus.cc:504] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.445851 2637 leader_election.cc:288] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 election: Requested vote from peers eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195), 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.446141 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d8082adb80497b81fbbb27aab6775d" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8bbe947a3457ba38eb19fdc57de96"
I20260619 16:35:40.446183 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d8082adb80497b81fbbb27aab6775d" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00"
I20260619 16:35:40.446287 2422 raft_consensus.cc:3037] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.446300 2562 raft_consensus.cc:3037] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.447316 2562 raft_consensus.cc:2454] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 1.
I20260619 16:35:40.447422 2422 raft_consensus.cc:2454] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 1.
I20260619 16:35:40.447593 2213 leader_election.cc:302] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eac8bbe947a3457ba38eb19fdc57de96, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:40.447762 2637 raft_consensus.cc:2781] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:40.447968 2637 raft_consensus.cc:686] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 1 LEADER]: Becoming Leader. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Running, Role: LEADER
I20260619 16:35:40.448268 2637 consensus_queue.cc:237] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } }
I20260619 16:35:40.451229 2124 catalog_manager.cc:5495] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d reported cstate change: term changed from 0 to 1, leader changed from <none> to efb7c6f830194368b1b6a11de3df809d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efb7c6f830194368b1b6a11de3df809d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:35:40 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:40.499143 2174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.491075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.0.98.62:33347)
0619 16:35:40.491219 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.491223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.491392 (+ 169us) server_negotiation.cc:407] Connection header received
0619 16:35:40.491514 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.491516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.491555 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.491636 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:40.492264 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.492811 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.493645 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.493858 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.493936 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.496082 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:40.496085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:40.496087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:40.496108 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:40.497563 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.497976 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.497979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.497980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.498064 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:40.498334 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:40.498337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:40.498339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:40.498522 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:40.498607 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.498904 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.499021 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":66}
I20260619 16:35:40.499143 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.491032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33347 (local address 127.0.0.1:33314)
0619 16:35:40.491303 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.491320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.491435 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.491685 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.491687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.492067 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:40.492163 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.492169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.492904 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.492906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.493549 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.493557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.493951 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.493954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.494010 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.494565 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:40.494576 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:40.495917 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:40.497659 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.497666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.497670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.497886 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.498144 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:40.498147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:40.498148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:40.498233 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:40.498637 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:40.498642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:40.498756 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.498850 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.498963 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
I20260619 16:35:40.503039 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33314:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:35:40.503757 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:40.508332 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 9340c0d0cd35427c80bfcf19dbcd6858 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52673da40e68463295165d22db0fd0b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:35:40.509095 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9340c0d0cd35427c80bfcf19dbcd6858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.508702 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 9340c0d0cd35427c80bfcf19dbcd6858 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52673da40e68463295165d22db0fd0b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:35:40.508800 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 9340c0d0cd35427c80bfcf19dbcd6858 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52673da40e68463295165d22db0fd0b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:35:40.509378 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9340c0d0cd35427c80bfcf19dbcd6858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.509522 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9340c0d0cd35427c80bfcf19dbcd6858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.511826 2633 tablet_bootstrap.cc:492] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: Bootstrap starting.
I20260619 16:35:40.512006 2631 tablet_bootstrap.cc:492] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: Bootstrap starting.
I20260619 16:35:40.512642 2633 tablet_bootstrap.cc:654] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.512806 2631 tablet_bootstrap.cc:654] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.513312 2632 tablet_bootstrap.cc:492] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: Bootstrap starting.
I20260619 16:35:40.513942 2631 tablet_bootstrap.cc:492] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: No bootstrap required, opened a new log
I20260619 16:35:40.514014 2631 ts_tablet_manager.cc:1397] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:35:40.514118 2633 tablet_bootstrap.cc:492] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: No bootstrap required, opened a new log
I20260619 16:35:40.514189 2633 ts_tablet_manager.cc:1397] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 16:35:40.514323 2632 tablet_bootstrap.cc:654] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.514426 2631 raft_consensus.cc:348] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.514505 2631 raft_consensus.cc:374] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.514525 2631 raft_consensus.cc:729] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a01c6775d541e08c342dc2a7f02c00, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.514645 2631 consensus_queue.cc:260] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.514680 2633 raft_consensus.cc:348] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.514782 2633 raft_consensus.cc:374] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.514819 2633 raft_consensus.cc:729] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.514927 2631 ts_tablet_manager.cc:1428] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:35:40.514878 2638 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.514997 2638 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.514946 2633 consensus_queue.cc:260] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.515218 2633 ts_tablet_manager.cc:1428] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:35:40.515538 2639 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.515671 2639 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.515988 2632 tablet_bootstrap.cc:492] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: No bootstrap required, opened a new log
I20260619 16:35:40.516043 2632 ts_tablet_manager.cc:1397] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:35:40.516364 2632 raft_consensus.cc:348] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.516436 2632 raft_consensus.cc:374] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.516456 2632 raft_consensus.cc:729] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.516577 2632 consensus_queue.cc:260] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.516842 2632 ts_tablet_manager.cc:1428] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:40.517014 2637 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.517117 2637 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.524019 2639 raft_consensus.cc:482] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:40.524134 2639 raft_consensus.cc:504] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.525095 2639 leader_election.cc:288] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919), 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.529245 2501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.525365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45919 (local address 127.0.98.3:43955)
0619 16:35:40.525569 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.525588 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.525686 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.526015 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.526020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.526041 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.526296 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.526303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.527234 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.527237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.528107 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.528116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.528798 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.528802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.528915 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.528949 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.529018 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.529098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":23,"spinlock_wait_cycles":17792}
I20260619 16:35:40.529260 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.525617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43955 (local address 127.0.98.1:45919)
0619 16:35:40.525767 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.525774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.525790 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:35:40.525836 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.525838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.525885 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.525988 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.526412 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.527140 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.528227 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.528693 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.528771 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.528815 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.529040 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.529125 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:35:40.529439 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.525632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40015 (local address 127.0.98.3:42053)
0619 16:35:40.525975 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.525990 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.526062 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.526292 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.526295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.526315 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:40.526448 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.526453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.527392 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.527395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.528322 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.528331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.529012 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.529016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.529111 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.529132 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.529208 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.529268 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":19,"thread_start_us":131,"threads_started":1}
I20260619 16:35:40.529453 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.525916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42053 (local address 127.0.98.2:40015)
0619 16:35:40.526036 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.526040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.526054 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:35:40.526160 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.526162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.526211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.526263 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:40.526573 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.527282 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.528437 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.528866 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.528966 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.529038 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.529237 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.529336 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:35:40.529947 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9340c0d0cd35427c80bfcf19dbcd6858" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00" is_pre_election: true
I20260619 16:35:40.529937 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9340c0d0cd35427c80bfcf19dbcd6858" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c6f830194368b1b6a11de3df809d" is_pre_election: true
I20260619 16:35:40.530076 2422 raft_consensus.cc:2454] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 0.
I20260619 16:35:40.530130 2282 raft_consensus.cc:2454] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 0.
I20260619 16:35:40.530417 2494 leader_election.cc:302] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a01c6775d541e08c342dc2a7f02c00, eac8bbe947a3457ba38eb19fdc57de96; no voters:
I20260619 16:35:40.530680 2639 raft_consensus.cc:2781] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:40.530735 2639 raft_consensus.cc:482] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:40.530761 2639 raft_consensus.cc:3037] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.531939 2639 raft_consensus.cc:504] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.532248 2639 leader_election.cc:288] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 election: Requested vote from peers efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919), 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.532624 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9340c0d0cd35427c80bfcf19dbcd6858" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c6f830194368b1b6a11de3df809d"
I20260619 16:35:40.532744 2282 raft_consensus.cc:3037] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.532801 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9340c0d0cd35427c80bfcf19dbcd6858" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00"
I20260619 16:35:40.532938 2422 raft_consensus.cc:3037] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.533735 2282 raft_consensus.cc:2454] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 1.
I20260619 16:35:40.533885 2422 raft_consensus.cc:2454] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 1.
I20260619 16:35:40.534040 2494 leader_election.cc:302] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eac8bbe947a3457ba38eb19fdc57de96, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:40.534202 2639 raft_consensus.cc:2781] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:40.534411 2639 raft_consensus.cc:686] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [term 1 LEADER]: Becoming Leader. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Running, Role: LEADER
I20260619 16:35:40.534719 2639 consensus_queue.cc:237] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.536733 2654 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: TxnStatusTablet state changed. Reason: New leader eac8bbe947a3457ba38eb19fdc57de96. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.536859 2654 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:35:40.537390 2656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:35:40.537607 2656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 16:35:40.537931 2124 catalog_manager.cc:5495] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 reported cstate change: term changed from 0 to 1, leader changed from <none> to eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } health_report { overall_health: HEALTHY } } }
I20260619 16:35:40.553386 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.549603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36195 (local address 127.0.0.1:42322)
0619 16:35:40.549722 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.549739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.549835 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.550135 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.550137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.550142 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:40.550233 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.550238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.551222 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.551224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.551796 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.551803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.552334 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.552336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.552399 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.552436 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.553110 (+ 674us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.553121 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.553168 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.553237 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 16:35:40.553392 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.549666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.0.98.3:36195)
0619 16:35:40.549793 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.549797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.549811 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:35:40.549959 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.549962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.550021 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.550123 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:40.550331 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.551104 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.551896 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.552231 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.552332 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.552556 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.553016 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.553128 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.553193 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.553243 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:35:40.557874 2422 raft_consensus.cc:1261] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Refusing update from remote peer eac8bbe947a3457ba38eb19fdc57de96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:40.557884 2282 raft_consensus.cc:1261] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Refusing update from remote peer eac8bbe947a3457ba38eb19fdc57de96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:40.558496 2639 consensus_queue.cc:1035] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:40.558668 2654 consensus_queue.cc:1035] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:40.561579 2637 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: TxnStatusTablet state changed. Reason: New leader eac8bbe947a3457ba38eb19fdc57de96. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.561685 2637 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.562196 2638 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: TxnStatusTablet state changed. Reason: New leader eac8bbe947a3457ba38eb19fdc57de96. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.562309 2638 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.565542 2639 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.565653 2639 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:35:40.566182 2638 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.566308 2638 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P 08a01c6775d541e08c342dc2a7f02c00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.566289 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298608908518989824 to 7298608908432777216. Current Snapshot: MvccSnapshot[applied={T|T < 7298608908518989824}]
I20260619 16:35:40.567363 2654 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.567456 2654 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P eac8bbe947a3457ba38eb19fdc57de96: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:35:40.568178 2637 tablet_replica.cc:439] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } } }
I20260619 16:35:40.568277 2637 tablet_replica.cc:442] T 9340c0d0cd35427c80bfcf19dbcd6858 P efb7c6f830194368b1b6a11de3df809d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:35:40.576669 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.572638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.0.98.1:45919)
0619 16:35:40.573021 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.573024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.573033 (+ 9us) server_negotiation.cc:407] Connection header received
0619 16:35:40.573081 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.573084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.573124 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.573238 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:40.573640 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.574205 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.575427 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.575634 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.575721 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.576001 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.576350 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.576430 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.576532 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.576574 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260619 16:35:40.587838 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.584153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36195 (local address 127.0.98.1:43093)
0619 16:35:40.584440 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.584453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.584533 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.584762 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.584764 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.584782 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:40.584871 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.584875 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.585919 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.585921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.586570 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.586579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.587152 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.587155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.587233 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.587271 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.587596 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.587601 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.587653 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.587712 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
I20260619 16:35:40.587853 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.584247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43093 (local address 127.0.98.3:36195)
0619 16:35:40.584362 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.584367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.584520 (+ 153us) server_negotiation.cc:407] Connection header received
0619 16:35:40.584603 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.584606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.584668 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.584754 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:40.584967 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.585825 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.586677 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.587031 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.587134 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.587415 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.587513 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.587586 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.587677 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.587722 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:35:40.592772 2609 tablet.cc:2369] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:40.594658 2422 raft_consensus.cc:1261] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Refusing update from remote peer efb7c6f830194368b1b6a11de3df809d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:40.594933 2562 raft_consensus.cc:1261] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Refusing update from remote peer efb7c6f830194368b1b6a11de3df809d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:40.595458 2637 consensus_queue.cc:1035] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:40.595654 2642 consensus_queue.cc:1035] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:40.599098 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298608908671545344 to 7298608908078600192. Current Snapshot: MvccSnapshot[applied={T|T < 7298608908671545344}]
I20260619 16:35:40.600555 2661 mvcc.cc:205] Tried to move back new op lower bound from 7298608908671545344 to 7298608908078600192. Current Snapshot: MvccSnapshot[applied={T|T < 7298608908671545344}]
I20260619 16:35:40.617712 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.614252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59835 (local address 127.0.98.1:45919)
0619 16:35:40.614361 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.614365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.614381 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:35:40.614507 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.614510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.614558 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.614636 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:40.614903 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.615695 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.616598 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.616939 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.617045 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.617270 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.617372 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.617444 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.617536 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.617584 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:35:40.617743 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.614186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45919 (local address 127.0.98.3:59835)
0619 16:35:40.614329 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.614344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.614418 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.614658 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.614661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.614671 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:40.614797 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.614805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.615872 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.615875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.616492 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.616500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.617045 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.617048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.617113 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.617152 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.617445 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.617449 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.617519 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.617600 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:35:40.639770 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:40.639948 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:40.642470 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Sending DeleteTablet for 3 replicas of tablet 63d8082adb80497b81fbbb27aab6775d
I20260619 16:35:40.643079 2542 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d8082adb80497b81fbbb27aab6775d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260619 16:35:40.643534 2678 tablet_replica.cc:330] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: stopping tablet replica
I20260619 16:35:40.643534 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d8082adb80497b81fbbb27aab6775d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60848
I20260619 16:35:40.643841 2678 raft_consensus.cc:2229] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:40.643889 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet 63d8082adb80497b81fbbb27aab6775d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39566
I20260619 16:35:40.643993 2678 raft_consensus.cc:2258] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:40.645073 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:40.645474 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:40.645591 2678 ts_tablet_manager.cc:1905] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:40.648631 2679 tablet_replica.cc:330] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: stopping tablet replica
I20260619 16:35:40.648874 2680 tablet_replica.cc:330] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: stopping tablet replica
I20260619 16:35:40.649191 2678 ts_tablet_manager.cc:1918] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:40.649287 2678 log.cc:1196] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/63d8082adb80497b81fbbb27aab6775d
I20260619 16:35:40.649561 2680 raft_consensus.cc:2229] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:40.649643 2678 ts_tablet_manager.cc:1939] T 63d8082adb80497b81fbbb27aab6775d P eac8bbe947a3457ba38eb19fdc57de96: Deleting consensus metadata
I20260619 16:35:40.649216 2679 raft_consensus.cc:2229] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:40.650163 2679 raft_consensus.cc:2258] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:40.650497 2680 raft_consensus.cc:2258] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:40.651016 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 3acce204b7e44f8db966e939c5dc676e (DEFAULT_TABLE table=test-table [id=d0256796239d4374a342c93f2fa9afc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:40.651252 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acce204b7e44f8db966e939c5dc676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.653398 2632 tablet_bootstrap.cc:492] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Bootstrap starting.
I20260619 16:35:40.653558 2542 tablet_service.cc:1467] Processing CreateTablet for tablet 3acce204b7e44f8db966e939c5dc676e (DEFAULT_TABLE table=test-table [id=d0256796239d4374a342c93f2fa9afc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:40.653796 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acce204b7e44f8db966e939c5dc676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.654332 2632 tablet_bootstrap.cc:654] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.654927 2109 catalog_manager.cc:4841] TS eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195): tablet 63d8082adb80497b81fbbb27aab6775d (table test-table [id=82bbed1fc62d4d5eb0f505d71e3ca8ec]) successfully deleted
I20260619 16:35:40.655824 2633 tablet_bootstrap.cc:492] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Bootstrap starting.
I20260619 16:35:40.655857 2679 ts_tablet_manager.cc:1905] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:40.656002 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 3acce204b7e44f8db966e939c5dc676e (DEFAULT_TABLE table=test-table [id=d0256796239d4374a342c93f2fa9afc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:40.656239 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3acce204b7e44f8db966e939c5dc676e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:40.656666 2633 tablet_bootstrap.cc:654] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.658716 2631 tablet_bootstrap.cc:492] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Bootstrap starting.
I20260619 16:35:40.658773 2679 ts_tablet_manager.cc:1918] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:40.658845 2679 log.cc:1196] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/63d8082adb80497b81fbbb27aab6775d
I20260619 16:35:40.659097 2679 ts_tablet_manager.cc:1939] T 63d8082adb80497b81fbbb27aab6775d P 08a01c6775d541e08c342dc2a7f02c00: Deleting consensus metadata
I20260619 16:35:40.659559 2631 tablet_bootstrap.cc:654] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:40.660324 2110 catalog_manager.cc:4841] TS 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015): tablet 63d8082adb80497b81fbbb27aab6775d (table test-table [id=82bbed1fc62d4d5eb0f505d71e3ca8ec]) successfully deleted
I20260619 16:35:40.660673 2633 tablet_bootstrap.cc:492] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: No bootstrap required, opened a new log
I20260619 16:35:40.660763 2633 ts_tablet_manager.cc:1397] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260619 16:35:40.660902 2632 tablet_bootstrap.cc:492] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: No bootstrap required, opened a new log
I20260619 16:35:40.660974 2632 ts_tablet_manager.cc:1397] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260619 16:35:40.661149 2633 raft_consensus.cc:348] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.661258 2633 raft_consensus.cc:374] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.661293 2633 raft_consensus.cc:729] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.661429 2633 consensus_queue.cc:260] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.661445 2632 raft_consensus.cc:348] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.661548 2632 raft_consensus.cc:374] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.661586 2632 raft_consensus.cc:729] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.661695 2633 ts_tablet_manager.cc:1428] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:40.661687 2632 consensus_queue.cc:260] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.661926 2632 ts_tablet_manager.cc:1428] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:40.665946 2631 tablet_bootstrap.cc:492] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: No bootstrap required, opened a new log
I20260619 16:35:40.666039 2631 ts_tablet_manager.cc:1397] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 16:35:40.666333 2680 ts_tablet_manager.cc:1905] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:40.666461 2631 raft_consensus.cc:348] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.666575 2631 raft_consensus.cc:374] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:40.666613 2631 raft_consensus.cc:729] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a01c6775d541e08c342dc2a7f02c00, State: Initialized, Role: FOLLOWER
I20260619 16:35:40.666746 2631 consensus_queue.cc:260] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.666994 2631 ts_tablet_manager.cc:1428] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:40.669515 2680 ts_tablet_manager.cc:1918] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:40.669602 2680 log.cc:1196] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/63d8082adb80497b81fbbb27aab6775d
I20260619 16:35:40.669881 2680 ts_tablet_manager.cc:1939] T 63d8082adb80497b81fbbb27aab6775d P efb7c6f830194368b1b6a11de3df809d: Deleting consensus metadata
I20260619 16:35:40.670847 2110 catalog_manager.cc:4841] TS efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919): tablet 63d8082adb80497b81fbbb27aab6775d (table test-table [id=82bbed1fc62d4d5eb0f505d71e3ca8ec]) successfully deleted
I20260619 16:35:40.783336 2639 raft_consensus.cc:482] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:40.783483 2639 raft_consensus.cc:504] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.783916 2639 leader_election.cc:288] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919), 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.784373 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce204b7e44f8db966e939c5dc676e" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c6f830194368b1b6a11de3df809d" is_pre_election: true
I20260619 16:35:40.784428 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce204b7e44f8db966e939c5dc676e" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00" is_pre_election: true
I20260619 16:35:40.784530 2282 raft_consensus.cc:2454] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 0.
I20260619 16:35:40.784602 2421 raft_consensus.cc:2454] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 0.
I20260619 16:35:40.784945 2494 leader_election.cc:302] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a01c6775d541e08c342dc2a7f02c00, eac8bbe947a3457ba38eb19fdc57de96; no voters:
I20260619 16:35:40.785116 2639 raft_consensus.cc:2781] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:40.785175 2639 raft_consensus.cc:482] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:40.785216 2639 raft_consensus.cc:3037] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.786258 2639 raft_consensus.cc:504] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.786640 2639 leader_election.cc:288] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 election: Requested vote from peers efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919), 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015)
I20260619 16:35:40.787009 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce204b7e44f8db966e939c5dc676e" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00"
I20260619 16:35:40.787020 2282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3acce204b7e44f8db966e939c5dc676e" candidate_uuid: "eac8bbe947a3457ba38eb19fdc57de96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c6f830194368b1b6a11de3df809d"
I20260619 16:35:40.787137 2421 raft_consensus.cc:3037] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.787155 2282 raft_consensus.cc:3037] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:40.788033 2282 raft_consensus.cc:2454] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 1.
I20260619 16:35:40.788306 2494 leader_election.cc:302] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eac8bbe947a3457ba38eb19fdc57de96, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:40.788342 2421 raft_consensus.cc:2454] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac8bbe947a3457ba38eb19fdc57de96 in term 1.
I20260619 16:35:40.788512 2639 raft_consensus.cc:2781] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:40.788595 2639 raft_consensus.cc:686] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 1 LEADER]: Becoming Leader. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Running, Role: LEADER
I20260619 16:35:40.788755 2639 consensus_queue.cc:237] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:40.790587 2123 catalog_manager.cc:5495] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 reported cstate change: term changed from 0 to 1, leader changed from <none> to eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "eac8bbe947a3457ba38eb19fdc57de96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } health_report { overall_health: HEALTHY } } }
I20260619 16:35:40.817071 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.813728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.0.98.3:36195)
0619 16:35:40.813855 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.813860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.813906 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:35:40.814009 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.814012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.814059 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.814134 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:40.814329 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.815097 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.815909 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.816257 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.816350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.816563 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.816655 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.816724 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.816888 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.816936 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:35:40.825424 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.822423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32861 (local address 127.0.98.3:36195)
0619 16:35:40.822498 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:40.822502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:40.822600 (+ 98us) server_negotiation.cc:407] Connection header received
0619 16:35:40.822691 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:40.822693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:40.822730 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:40.822781 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:40.822996 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.823450 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.824503 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.824767 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.824846 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.825030 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.825083 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.825160 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:40.825254 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:35:40.825295 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":24}
I20260619 16:35:40.825455 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:40.822348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36195 (local address 127.0.98.3:32861)
0619 16:35:40.822524 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:40.822538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:40.822614 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:40.822811 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:40.822813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:40.822820 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:35:40.822897 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.822902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.823556 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.823559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.824384 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:40.824395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:40.824863 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:40.824866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:40.824920 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:40.824941 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:35:40.825178 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:35:40.825182 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:35:40.825244 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:35:40.825300 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260619 16:35:40.831017 2281 raft_consensus.cc:1261] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Refusing update from remote peer eac8bbe947a3457ba38eb19fdc57de96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:40.831173 2421 raft_consensus.cc:1261] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Refusing update from remote peer eac8bbe947a3457ba38eb19fdc57de96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:40.831426 2654 consensus_queue.cc:1035] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:40.831562 2666 consensus_queue.cc:1035] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:35:40.845424 2659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:35:42.403904 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:42.404073 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:42.406562 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Sending DeleteTablet for 3 replicas of tablet 3acce204b7e44f8db966e939c5dc676e
I20260619 16:35:42.407074 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acce204b7e44f8db966e939c5dc676e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39566
I20260619 16:35:42.407193 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acce204b7e44f8db966e939c5dc676e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260619 16:35:42.407214 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 3acce204b7e44f8db966e939c5dc676e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60848
I20260619 16:35:42.407486 2708 tablet_replica.cc:330] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: stopping tablet replica
I20260619 16:35:42.407611 2708 raft_consensus.cc:2229] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:42.407717 2708 raft_consensus.cc:2258] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:42.408792 2708 ts_tablet_manager.cc:1905] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:42.410283 2709 tablet_replica.cc:330] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: stopping tablet replica
I20260619 16:35:42.410516 2709 raft_consensus.cc:2229] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:42.410874 2709 raft_consensus.cc:2258] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:42.411449 2709 ts_tablet_manager.cc:1905] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:42.411763 2708 ts_tablet_manager.cc:1918] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:35:42.411844 2708 log.cc:1196] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/3acce204b7e44f8db966e939c5dc676e
I20260619 16:35:42.412099 2708 ts_tablet_manager.cc:1939] T 3acce204b7e44f8db966e939c5dc676e P efb7c6f830194368b1b6a11de3df809d: Deleting consensus metadata
I20260619 16:35:42.412837 2110 catalog_manager.cc:4841] TS efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919): tablet 3acce204b7e44f8db966e939c5dc676e (table test-table [id=d0256796239d4374a342c93f2fa9afc2]) successfully deleted
I20260619 16:35:42.413502 2710 tablet_replica.cc:330] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: stopping tablet replica
I20260619 16:35:42.413664 2710 raft_consensus.cc:2229] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:42.413794 2710 raft_consensus.cc:2258] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:42.414381 2710 ts_tablet_manager.cc:1905] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:42.414508 2709 ts_tablet_manager.cc:1918] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:35:42.414600 2709 log.cc:1196] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/3acce204b7e44f8db966e939c5dc676e
I20260619 16:35:42.414942 2709 ts_tablet_manager.cc:1939] T 3acce204b7e44f8db966e939c5dc676e P eac8bbe947a3457ba38eb19fdc57de96: Deleting consensus metadata
I20260619 16:35:42.415657 2109 catalog_manager.cc:4841] TS eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195): tablet 3acce204b7e44f8db966e939c5dc676e (table test-table [id=d0256796239d4374a342c93f2fa9afc2]) successfully deleted
I20260619 16:35:42.416591 2710 ts_tablet_manager.cc:1918] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:35:42.416680 2710 log.cc:1196] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/3acce204b7e44f8db966e939c5dc676e
I20260619 16:35:42.416896 2710 ts_tablet_manager.cc:1939] T 3acce204b7e44f8db966e939c5dc676e P 08a01c6775d541e08c342dc2a7f02c00: Deleting consensus metadata
I20260619 16:35:42.417479 2110 catalog_manager.cc:4841] TS 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015): tablet 3acce204b7e44f8db966e939c5dc676e (table test-table [id=d0256796239d4374a342c93f2fa9afc2]) successfully deleted
Jun 19 16:35:42 dist-test-slave-0f28 krb5kdc[2080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886942, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:35:42 dist-test-slave-0f28 krb5kdc[2080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886942, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:42.436327 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:42.427428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.0.98.62:33347)
0619 16:35:42.427882 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:42.427890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:42.427904 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:35:42.428035 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:42.428038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:42.428094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:42.428189 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:42.428852 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:42.429582 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:42.430409 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:42.430663 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:42.430744 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:42.433246 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:42.433249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:42.433251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:42.433274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:42.434751 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:42.435119 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:42.435121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:42.435122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:42.435195 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:42.435452 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:42.435456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:42.435457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:42.435657 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:42.435774 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:42.435993 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:35:42.436164 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":167,"threads_started":1}
I20260619 16:35:42.438973 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:42.439406 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:42.444161 2262 tablet_service.cc:1467] Processing CreateTablet for tablet 596f189b4f524dd48c5e6a4299c46d7f (DEFAULT_TABLE table=test-table [id=93f1c78929a24800ae5c59f8bd236639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:42.444168 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 596f189b4f524dd48c5e6a4299c46d7f (DEFAULT_TABLE table=test-table [id=93f1c78929a24800ae5c59f8bd236639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:42.444321 2541 tablet_service.cc:1467] Processing CreateTablet for tablet 596f189b4f524dd48c5e6a4299c46d7f (DEFAULT_TABLE table=test-table [id=93f1c78929a24800ae5c59f8bd236639]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:42.444466 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596f189b4f524dd48c5e6a4299c46d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:42.444471 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596f189b4f524dd48c5e6a4299c46d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:42.444599 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596f189b4f524dd48c5e6a4299c46d7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:42.446745 2719 tablet_bootstrap.cc:492] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Bootstrap starting.
I20260619 16:35:42.447356 2721 tablet_bootstrap.cc:492] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Bootstrap starting.
I20260619 16:35:42.447542 2720 tablet_bootstrap.cc:492] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Bootstrap starting.
I20260619 16:35:42.448125 2719 tablet_bootstrap.cc:654] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:42.448221 2721 tablet_bootstrap.cc:654] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:42.448439 2720 tablet_bootstrap.cc:654] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:42.449668 2721 tablet_bootstrap.cc:492] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: No bootstrap required, opened a new log
I20260619 16:35:42.449757 2721 ts_tablet_manager.cc:1397] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:42.449824 2719 tablet_bootstrap.cc:492] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: No bootstrap required, opened a new log
I20260619 16:35:42.449928 2719 ts_tablet_manager.cc:1397] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:35:42.450196 2721 raft_consensus.cc:348] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.450307 2721 raft_consensus.cc:374] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:42.450366 2721 raft_consensus.cc:729] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Initialized, Role: FOLLOWER
I20260619 16:35:42.450399 2719 raft_consensus.cc:348] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.450529 2719 raft_consensus.cc:374] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:42.450579 2719 raft_consensus.cc:729] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a01c6775d541e08c342dc2a7f02c00, State: Initialized, Role: FOLLOWER
I20260619 16:35:42.450538 2721 consensus_queue.cc:260] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.450711 2719 consensus_queue.cc:260] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.450851 2721 ts_tablet_manager.cc:1428] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:35:42.451105 2719 ts_tablet_manager.cc:1428] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:35:42.451736 2720 tablet_bootstrap.cc:492] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: No bootstrap required, opened a new log
I20260619 16:35:42.451830 2720 ts_tablet_manager.cc:1397] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:35:42.452277 2720 raft_consensus.cc:348] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.452416 2720 raft_consensus.cc:374] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:42.452468 2720 raft_consensus.cc:729] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Initialized, Role: FOLLOWER
I20260619 16:35:42.452652 2720 consensus_queue.cc:260] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.453043 2720 ts_tablet_manager.cc:1428] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:35:42.577337 2726 raft_consensus.cc:482] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:42.577517 2726 raft_consensus.cc:504] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.577963 2726 leader_election.cc:288] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015), eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195)
I20260619 16:35:42.578444 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596f189b4f524dd48c5e6a4299c46d7f" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00" is_pre_election: true
I20260619 16:35:42.578442 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596f189b4f524dd48c5e6a4299c46d7f" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8bbe947a3457ba38eb19fdc57de96" is_pre_election: true
I20260619 16:35:42.578605 2421 raft_consensus.cc:2454] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 0.
I20260619 16:35:42.578605 2562 raft_consensus.cc:2454] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 0.
I20260619 16:35:42.578948 2213 leader_election.cc:302] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eac8bbe947a3457ba38eb19fdc57de96, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:42.579156 2726 raft_consensus.cc:2781] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:42.579213 2726 raft_consensus.cc:482] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:42.579262 2726 raft_consensus.cc:3037] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:42.580256 2726 raft_consensus.cc:504] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.580581 2726 leader_election.cc:288] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 election: Requested vote from peers 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015), eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195)
I20260619 16:35:42.580950 2421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596f189b4f524dd48c5e6a4299c46d7f" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00"
I20260619 16:35:42.580969 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596f189b4f524dd48c5e6a4299c46d7f" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8bbe947a3457ba38eb19fdc57de96"
I20260619 16:35:42.581055 2421 raft_consensus.cc:3037] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:42.581110 2562 raft_consensus.cc:3037] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:42.582003 2421 raft_consensus.cc:2454] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 1.
I20260619 16:35:42.582053 2562 raft_consensus.cc:2454] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 1.
I20260619 16:35:42.582273 2214 leader_election.cc:302] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a01c6775d541e08c342dc2a7f02c00, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:42.582459 2726 raft_consensus.cc:2781] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:42.582676 2726 raft_consensus.cc:686] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 1 LEADER]: Becoming Leader. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Running, Role: LEADER
I20260619 16:35:42.582826 2726 consensus_queue.cc:237] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.584340 2124 catalog_manager.cc:5495] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d reported cstate change: term changed from 0 to 1, leader changed from <none> to efb7c6f830194368b1b6a11de3df809d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efb7c6f830194368b1b6a11de3df809d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } health_report { overall_health: UNKNOWN } } }
I20260619 16:35:42.607379 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:42.604315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.0.98.1:45919)
0619 16:35:42.604567 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:42.604573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:42.604605 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:35:42.604667 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:42.604670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:42.604720 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:42.604790 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:42.605021 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:42.605643 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:42.606479 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:42.606712 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:42.606791 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:42.607020 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:42.607084 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:42.607136 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:42.607230 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 16:35:42.607271 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":124,"threads_started":1}
W20260619 16:35:42.609283 2469 tablet.cc:2369] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:42.612913 2562 raft_consensus.cc:1261] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Refusing update from remote peer efb7c6f830194368b1b6a11de3df809d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:42.612939 2421 raft_consensus.cc:1261] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Refusing update from remote peer efb7c6f830194368b1b6a11de3df809d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:42.613329 2727 consensus_queue.cc:1035] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:42.613464 2726 consensus_queue.cc:1035] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:42.642390 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:42.642519 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:42.644302 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Sending DeleteTablet for 3 replicas of tablet 596f189b4f524dd48c5e6a4299c46d7f
I20260619 16:35:42.644794 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet 596f189b4f524dd48c5e6a4299c46d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39566
I20260619 16:35:42.644897 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 596f189b4f524dd48c5e6a4299c46d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60848
I20260619 16:35:42.644990 2708 tablet_replica.cc:330] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: stopping tablet replica
I20260619 16:35:42.644985 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet 596f189b4f524dd48c5e6a4299c46d7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260619 16:35:42.645119 2708 raft_consensus.cc:2229] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:42.645160 2709 tablet_replica.cc:330] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: stopping tablet replica
I20260619 16:35:42.645179 2710 tablet_replica.cc:330] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: stopping tablet replica
I20260619 16:35:42.645251 2709 raft_consensus.cc:2229] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:42.645285 2708 raft_consensus.cc:2258] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:42.645298 2710 raft_consensus.cc:2229] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:42.645370 2709 raft_consensus.cc:2258] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:42.645431 2710 raft_consensus.cc:2258] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:42.645912 2708 ts_tablet_manager.cc:1905] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:42.646085 2709 ts_tablet_manager.cc:1905] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:42.646004 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:42.646384 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:42.648384 2708 ts_tablet_manager.cc:1918] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:42.648408 2709 ts_tablet_manager.cc:1918] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:42.648471 2708 log.cc:1196] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/596f189b4f524dd48c5e6a4299c46d7f
I20260619 16:35:42.648464 2709 log.cc:1196] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/596f189b4f524dd48c5e6a4299c46d7f
I20260619 16:35:42.648716 2709 ts_tablet_manager.cc:1939] T 596f189b4f524dd48c5e6a4299c46d7f P eac8bbe947a3457ba38eb19fdc57de96: Deleting consensus metadata
I20260619 16:35:42.648718 2708 ts_tablet_manager.cc:1939] T 596f189b4f524dd48c5e6a4299c46d7f P efb7c6f830194368b1b6a11de3df809d: Deleting consensus metadata
I20260619 16:35:42.649415 2109 catalog_manager.cc:4841] TS eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195): tablet 596f189b4f524dd48c5e6a4299c46d7f (table test-table [id=93f1c78929a24800ae5c59f8bd236639]) successfully deleted
I20260619 16:35:42.649600 2110 catalog_manager.cc:4841] TS efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919): tablet 596f189b4f524dd48c5e6a4299c46d7f (table test-table [id=93f1c78929a24800ae5c59f8bd236639]) successfully deleted
I20260619 16:35:42.649735 2710 ts_tablet_manager.cc:1905] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:42.651854 2710 ts_tablet_manager.cc:1918] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:35:42.651818 2541 tablet_service.cc:1467] Processing CreateTablet for tablet f1b172a642084507942c36fa56fbbf0b (DEFAULT_TABLE table=test-table [id=f8e42ea6173f4306af805b1cb3b9c67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:42.651940 2710 log.cc:1196] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/596f189b4f524dd48c5e6a4299c46d7f
I20260619 16:35:42.652025 2541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b172a642084507942c36fa56fbbf0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:42.652206 2710 ts_tablet_manager.cc:1939] T 596f189b4f524dd48c5e6a4299c46d7f P 08a01c6775d541e08c342dc2a7f02c00: Deleting consensus metadata
I20260619 16:35:42.653734 2110 catalog_manager.cc:4841] TS 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015): tablet 596f189b4f524dd48c5e6a4299c46d7f (table test-table [id=93f1c78929a24800ae5c59f8bd236639]) successfully deleted
I20260619 16:35:42.653880 2262 tablet_service.cc:1467] Processing CreateTablet for tablet f1b172a642084507942c36fa56fbbf0b (DEFAULT_TABLE table=test-table [id=f8e42ea6173f4306af805b1cb3b9c67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:42.654016 2721 tablet_bootstrap.cc:492] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Bootstrap starting.
I20260619 16:35:42.654126 2262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b172a642084507942c36fa56fbbf0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:42.654801 2721 tablet_bootstrap.cc:654] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:42.655323 2402 tablet_service.cc:1467] Processing CreateTablet for tablet f1b172a642084507942c36fa56fbbf0b (DEFAULT_TABLE table=test-table [id=f8e42ea6173f4306af805b1cb3b9c67f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:42.655550 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b172a642084507942c36fa56fbbf0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:42.656024 2720 tablet_bootstrap.cc:492] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d: Bootstrap starting.
I20260619 16:35:42.656816 2720 tablet_bootstrap.cc:654] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:42.657534 2719 tablet_bootstrap.cc:492] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Bootstrap starting.
I20260619 16:35:42.657903 2721 tablet_bootstrap.cc:492] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: No bootstrap required, opened a new log
I20260619 16:35:42.657967 2721 ts_tablet_manager.cc:1397] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:35:42.658363 2719 tablet_bootstrap.cc:654] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:42.658319 2721 raft_consensus.cc:348] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.658427 2721 raft_consensus.cc:374] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:42.658474 2721 raft_consensus.cc:729] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac8bbe947a3457ba38eb19fdc57de96, State: Initialized, Role: FOLLOWER
I20260619 16:35:42.658605 2721 consensus_queue.cc:260] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.658874 2721 ts_tablet_manager.cc:1428] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:42.659268 2720 tablet_bootstrap.cc:492] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d: No bootstrap required, opened a new log
I20260619 16:35:42.659375 2720 ts_tablet_manager.cc:1397] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:35:42.659729 2720 raft_consensus.cc:348] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.659833 2720 raft_consensus.cc:374] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:42.659871 2720 raft_consensus.cc:729] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Initialized, Role: FOLLOWER
I20260619 16:35:42.659996 2720 consensus_queue.cc:260] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.660260 2720 ts_tablet_manager.cc:1428] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:35:42.660946 2719 tablet_bootstrap.cc:492] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: No bootstrap required, opened a new log
I20260619 16:35:42.661042 2719 ts_tablet_manager.cc:1397] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 16:35:42.661500 2719 raft_consensus.cc:348] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.661612 2719 raft_consensus.cc:374] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:42.661655 2719 raft_consensus.cc:729] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08a01c6775d541e08c342dc2a7f02c00, State: Initialized, Role: FOLLOWER
I20260619 16:35:42.661782 2719 consensus_queue.cc:260] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.662057 2719 ts_tablet_manager.cc:1428] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:35:42.834772 2726 raft_consensus.cc:482] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:42.834916 2726 raft_consensus.cc:504] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.835429 2726 leader_election.cc:288] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015), eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195)
I20260619 16:35:42.835929 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b172a642084507942c36fa56fbbf0b" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00" is_pre_election: true
I20260619 16:35:42.835930 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b172a642084507942c36fa56fbbf0b" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8bbe947a3457ba38eb19fdc57de96" is_pre_election: true
I20260619 16:35:42.836102 2422 raft_consensus.cc:2454] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 0.
I20260619 16:35:42.836113 2562 raft_consensus.cc:2454] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 0.
I20260619 16:35:42.836419 2214 leader_election.cc:302] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a01c6775d541e08c342dc2a7f02c00, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:42.836621 2726 raft_consensus.cc:2781] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:42.836685 2726 raft_consensus.cc:482] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:42.836715 2726 raft_consensus.cc:3037] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:42.837723 2726 raft_consensus.cc:504] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.838039 2726 leader_election.cc:288] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 election: Requested vote from peers 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015), eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195)
I20260619 16:35:42.838421 2422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b172a642084507942c36fa56fbbf0b" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08a01c6775d541e08c342dc2a7f02c00"
I20260619 16:35:42.838438 2562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b172a642084507942c36fa56fbbf0b" candidate_uuid: "efb7c6f830194368b1b6a11de3df809d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac8bbe947a3457ba38eb19fdc57de96"
I20260619 16:35:42.838519 2422 raft_consensus.cc:3037] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:42.838581 2562 raft_consensus.cc:3037] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:42.839461 2422 raft_consensus.cc:2454] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 1.
I20260619 16:35:42.839649 2562 raft_consensus.cc:2454] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c6f830194368b1b6a11de3df809d in term 1.
I20260619 16:35:42.839761 2214 leader_election.cc:302] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08a01c6775d541e08c342dc2a7f02c00, efb7c6f830194368b1b6a11de3df809d; no voters:
I20260619 16:35:42.839962 2726 raft_consensus.cc:2781] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:42.840085 2726 raft_consensus.cc:686] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [term 1 LEADER]: Becoming Leader. State: Replica: efb7c6f830194368b1b6a11de3df809d, State: Running, Role: LEADER
I20260619 16:35:42.840241 2726 consensus_queue.cc:237] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } }
I20260619 16:35:42.842031 2123 catalog_manager.cc:5495] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d reported cstate change: term changed from 0 to 1, leader changed from <none> to efb7c6f830194368b1b6a11de3df809d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efb7c6f830194368b1b6a11de3df809d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efb7c6f830194368b1b6a11de3df809d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 } health_report { overall_health: UNKNOWN } } }
I20260619 16:35:42.900931 2562 raft_consensus.cc:1261] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Refusing update from remote peer efb7c6f830194368b1b6a11de3df809d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:42.901333 2726 consensus_queue.cc:1035] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eac8bbe947a3457ba38eb19fdc57de96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:42.901438 2422 raft_consensus.cc:1261] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Refusing update from remote peer efb7c6f830194368b1b6a11de3df809d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:42.902415 2726 consensus_queue.cc:1035] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d [LEADER]: Connected to new peer: Peer: permanent_uuid: "08a01c6775d541e08c342dc2a7f02c00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:35:42.915160 2693 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 16:35:44.465731 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:44.465910 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:44.468344 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47fff942ff2b4eb0897443bdf4def88b: Sending DeleteTablet for 3 replicas of tablet f1b172a642084507942c36fa56fbbf0b
I20260619 16:35:44.469008 2262 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b172a642084507942c36fa56fbbf0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39566
I20260619 16:35:44.469077 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b172a642084507942c36fa56fbbf0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60848
I20260619 16:35:44.469117 2541 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b172a642084507942c36fa56fbbf0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42310
I20260619 16:35:44.469470 2762 tablet_replica.cc:330] T f1b172a642084507942c36fa56fbbf0b P efb7c6f830194368b1b6a11de3df809d: stopping tablet replica
I20260619 16:35:44.469485 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 2197
I20260619 16:35:44.471566 2764 tablet_replica.cc:330] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: stopping tablet replica
I20260619 16:35:44.471755 2764 raft_consensus.cc:2229] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:44.471917 2764 raft_consensus.cc:2258] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:44.472179 2763 tablet_replica.cc:330] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: stopping tablet replica
I20260619 16:35:44.472364 2763 raft_consensus.cc:2229] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:44.472527 2763 raft_consensus.cc:2258] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:44.472687 2764 ts_tablet_manager.cc:1905] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:44.473299 2763 ts_tablet_manager.cc:1905] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:44.475941 2764 ts_tablet_manager.cc:1918] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:35:44.476471 2764 log.cc:1196] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/f1b172a642084507942c36fa56fbbf0b
I20260619 16:35:44.476933 2764 ts_tablet_manager.cc:1939] T f1b172a642084507942c36fa56fbbf0b P eac8bbe947a3457ba38eb19fdc57de96: Deleting consensus metadata
I20260619 16:35:44.476996 2763 ts_tablet_manager.cc:1918] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 16:35:44.477100 2763 log.cc:1196] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/f1b172a642084507942c36fa56fbbf0b
I20260619 16:35:44.477418 2763 ts_tablet_manager.cc:1939] T f1b172a642084507942c36fa56fbbf0b P 08a01c6775d541e08c342dc2a7f02c00: Deleting consensus metadata
W20260619 16:35:44.477983 2110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45919 (error 108)
I20260619 16:35:44.479529 2110 catalog_manager.cc:4841] TS 08a01c6775d541e08c342dc2a7f02c00 (127.0.98.2:40015): tablet f1b172a642084507942c36fa56fbbf0b (table test-table [id=f8e42ea6173f4306af805b1cb3b9c67f]) successfully deleted
I20260619 16:35:44.479718 2109 catalog_manager.cc:4841] TS eac8bbe947a3457ba38eb19fdc57de96 (127.0.98.3:36195): tablet f1b172a642084507942c36fa56fbbf0b (table test-table [id=f8e42ea6173f4306af805b1cb3b9c67f]) successfully deleted
W20260619 16:35:44.480338 2766 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:44.479053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45919 (local address 127.0.0.1:37634)
0619 16:35:44.479543 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:44.479557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:44.479684 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:44.480232 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45919: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":322,"threads_started":1}
I20260619 16:35:44.480535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 2337
W20260619 16:35:44.480507 2110 catalog_manager.cc:4584] TS efb7c6f830194368b1b6a11de3df809d (127.0.98.1:45919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1b172a642084507942c36fa56fbbf0b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45919: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:35:44.486680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 2477
I20260619 16:35:44.493942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 2093
2026-06-19T16:35:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6193 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:44 dist-test-slave-0f28 krb5kdc[2769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:44 dist-test-slave-0f28 krb5kdc[2769](info): set up 2 sockets
Jun 19 16:35:44 dist-test-slave-0f28 krb5kdc[2769](info): commencing operation
krb5kdc: starting...
W20260619 16:35:45.557236 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:45 dist-test-slave-0f28 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:45.675963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:45.779395 2782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:45.779621 2782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:45.779659 2782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:45.783479 2782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:45.783524 2782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:45.783545 2782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:45.783565 2782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:45.783581 2782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:45.788492 2782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:45.789302 2782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:45.790169 2782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:45.798314 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:45.798331 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:45.798384 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:45.798419 2782 server_base.cc:1034] running on GCE node
I20260619 16:35:45.799094 2782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:45.800113 2782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:45.801306 2782 hybrid_clock.cc:648] HybridClock initialized: now 1781886945801294 us; error 27 us; skew 500 ppm
Jun 19 16:35:45 dist-test-slave-0f28 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:45.804028 2782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:45.804749 2782 webserver.cc:460] Webserver started at http://127.0.98.62:40797/ using document root <none> and password file <none>
I20260619 16:35:45.805212 2782 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:45.805267 2782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:45.805480 2782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:45.807281 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e"
format_stamp: "Formatted at 2026-06-19 16:35:45 on dist-test-slave-0f28"
server_key: "28012aa6a8f7e948e913a63e7b43be89"
server_key_iv: "98244be2bc6c6150a6c920f8db89c7dc"
server_key_version: "encryptionkey@0"
I20260619 16:35:45.807753 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e"
format_stamp: "Formatted at 2026-06-19 16:35:45 on dist-test-slave-0f28"
server_key: "28012aa6a8f7e948e913a63e7b43be89"
server_key_iv: "98244be2bc6c6150a6c920f8db89c7dc"
server_key_version: "encryptionkey@0"
I20260619 16:35:45.811295 2782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:45.813453 2797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:45.814594 2782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:45.814713 2782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e"
format_stamp: "Formatted at 2026-06-19 16:35:45 on dist-test-slave-0f28"
server_key: "28012aa6a8f7e948e913a63e7b43be89"
server_key_iv: "98244be2bc6c6150a6c920f8db89c7dc"
server_key_version: "encryptionkey@0"
I20260619 16:35:45.814808 2782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:45.826781 2782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:45.829794 2782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:45.829985 2782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:45.836637 2782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43529
I20260619 16:35:45.836724 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43529 every 8 connection(s)
I20260619 16:35:45.837639 2782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:45.840276 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:45.841213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 2782
I20260619 16:35:45.841351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:45.841646 392 external_mini_cluster.cc:1428] Setting key 022b008c82ddc362c3398c14516994a3
I20260619 16:35:45.846321 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e: Bootstrap starting.
I20260619 16:35:45.848838 2850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:45.849762 2850 log.cc:824] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:35:45 dist-test-slave-0f28 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:45.851634 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e: No bootstrap required, opened a new log
I20260619 16:35:45.855561 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:45.842939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.0.98.62:43529)
0619 16:35:45.843339 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:45.843353 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:45.843388 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:35:45.843985 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:45.844000 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:45.844303 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:45.844574 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:45.845457 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:45.846465 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:45.847625 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:45.848088 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:45.848209 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:45.851023 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:45.851028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:45.851040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:45.851063 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:45.852763 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:45.853242 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:45.853246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:45.853250 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:45.853418 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:45.853779 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:45.853782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:45.853783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:45.854147 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:45.854296 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:45.854576 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 16:35:45.854864 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":130,"threads_started":1}
I20260619 16:35:45.855633 2850 raft_consensus.cc:348] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260619 16:35:45.855927 2850 raft_consensus.cc:374] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:45.856005 2850 raft_consensus.cc:729] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d11b5a8a3c3c4e5faa361a5bc9af956e, State: Initialized, Role: FOLLOWER
I20260619 16:35:45.856410 2850 consensus_queue.cc:260] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260619 16:35:45.856566 2850 raft_consensus.cc:388] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:45.856635 2850 raft_consensus.cc:482] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:45.856750 2850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:45.857654 2850 raft_consensus.cc:504] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260619 16:35:45.857970 2850 leader_election.cc:302] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d11b5a8a3c3c4e5faa361a5bc9af956e; no voters:
I20260619 16:35:45.858251 2850 leader_election.cc:288] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:45.858352 2855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:45.858567 2855 raft_consensus.cc:686] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [term 1 LEADER]: Becoming Leader. State: Replica: d11b5a8a3c3c4e5faa361a5bc9af956e, State: Running, Role: LEADER
I20260619 16:35:45.858938 2855 consensus_queue.cc:237] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } }
I20260619 16:35:45.859439 2850 sys_catalog.cc:564] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:45.860548 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } } }
I20260619 16:35:45.860548 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d11b5a8a3c3c4e5faa361a5bc9af956e. Latest consensus state: current_term: 1 leader_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d11b5a8a3c3c4e5faa361a5bc9af956e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43529 } } }
I20260619 16:35:45.860682 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:45.860682 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:45.861089 2864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:45.864135 2864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:45.869524 2864 catalog_manager.cc:1347] Generated new cluster ID: 9bec9c935ca245ebaac9ab089427a650
I20260619 16:35:45.869582 2864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:45.877633 2864 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:45.878727 2864 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:45.884222 2864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d11b5a8a3c3c4e5faa361a5bc9af956e: Generated new TSK 0
I20260619 16:35:45.884889 2864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:45.912173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:46.018553 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:46.018810 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:46.018875 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:46.022837 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:46.022910 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:46.022994 2876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:46.027925 2876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:46.028861 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:46.029819 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:46.038921 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:46.038970 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:46.039004 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:46.039044 2876 server_base.cc:1034] running on GCE node
I20260619 16:35:46.039525 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:46.040120 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:46.041311 2876 hybrid_clock.cc:648] HybridClock initialized: now 1781886946041280 us; error 46 us; skew 500 ppm
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:46.043962 2876 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:46.044749 2876 webserver.cc:460] Webserver started at http://127.0.98.1:45447/ using document root <none> and password file <none>
I20260619 16:35:46.045284 2876 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:46.045403 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:46.045565 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:46.047464 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "1557d533cbbe48c6993bda87997842f6"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "75eaf4d07e9c124b531f6766ee5fea8d"
server_key_iv: "58dc1d632da39e0e6f1cf983794fe887"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.047942 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1557d533cbbe48c6993bda87997842f6"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "75eaf4d07e9c124b531f6766ee5fea8d"
server_key_iv: "58dc1d632da39e0e6f1cf983794fe887"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.051373 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:46.053592 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.054737 2876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:46.054870 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "1557d533cbbe48c6993bda87997842f6"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "75eaf4d07e9c124b531f6766ee5fea8d"
server_key_iv: "58dc1d632da39e0e6f1cf983794fe887"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.054977 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:46.077705 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:46.080675 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:46.080885 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:46.081596 2876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:46.082423 2876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:46.082468 2876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.082506 2876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:46.082522 2876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.091707 2876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43163
I20260619 16:35:46.091825 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43163 every 8 connection(s)
I20260619 16:35:46.092590 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:46.098042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 2876
I20260619 16:35:46.098132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:46.098416 392 external_mini_cluster.cc:1428] Setting key 5fc0defa54b6386179354d4cc475c0a7
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:46.106909 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:46.094401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43035 (local address 127.0.98.62:43529)
0619 16:35:46.094566 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:46.094569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:46.095333 (+ 764us) server_negotiation.cc:407] Connection header received
0619 16:35:46.096705 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:46.096709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:46.096767 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:46.096852 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:46.098005 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.098680 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.100183 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.100422 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.100544 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:46.103216 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:46.103219 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:46.103221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:46.103244 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:46.104786 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:46.105311 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:46.105315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:46.105352 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:46.105432 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:46.105854 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:46.105859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:46.105861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:46.106037 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:46.106141 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:46.106611 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 16:35:46.106741 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:35:46.107679 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:46.094757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43529 (local address 127.0.98.1:43035)
0619 16:35:46.095183 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:46.095224 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:46.096489 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:46.097026 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:46.097032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:46.097380 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:46.097857 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:46.097870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.098863 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.098867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:46.100060 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:46.100070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.100572 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.100576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:46.100745 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:46.101377 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:46.101404 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:46.103035 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:46.104917 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:46.104923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:46.104936 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:46.105198 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:46.105606 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:46.105610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:46.105612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:46.105757 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:46.106138 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:46.106144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:46.106372 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 16:35:46.106558 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 16:35:46.106857 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260619 16:35:46.108922 3005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43529
I20260619 16:35:46.109174 3005 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:46.109665 3005 heartbeater.cc:511] Master 127.0.98.62:43529 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:35:46.111078 2814 ts_manager.cc:194] Registered new tserver with Master: 1557d533cbbe48c6993bda87997842f6 (127.0.98.1:43163)
I20260619 16:35:46.112205 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:46.124838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:46.233606 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:46.233836 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:46.233875 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:46.237877 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:46.237923 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:46.237978 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:46.243021 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:46.243847 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:46.244771 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:46.253506 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:46.253520 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:46.253525 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:46.253829 3010 server_base.cc:1034] running on GCE node
I20260619 16:35:46.254172 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:46.254679 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:46.255883 3010 hybrid_clock.cc:648] HybridClock initialized: now 1781886946255871 us; error 23 us; skew 500 ppm
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:46.258558 3010 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:46.259356 3010 webserver.cc:460] Webserver started at http://127.0.98.2:45983/ using document root <none> and password file <none>
I20260619 16:35:46.259965 3010 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:46.260067 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:46.260277 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:46.262231 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "079d5ed2a3804feaacad121e6cdd899f"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "fde36b5b3bf6c00b8f84953afc6764d1"
server_key_iv: "91dfc598647896ed68dec5b0f65e130e"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.262805 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "079d5ed2a3804feaacad121e6cdd899f"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "fde36b5b3bf6c00b8f84953afc6764d1"
server_key_iv: "91dfc598647896ed68dec5b0f65e130e"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.266458 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:46.268755 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.270216 3010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:46.270340 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "079d5ed2a3804feaacad121e6cdd899f"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "fde36b5b3bf6c00b8f84953afc6764d1"
server_key_iv: "91dfc598647896ed68dec5b0f65e130e"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.270435 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:46.296834 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:46.299626 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:46.299814 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:46.300403 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:46.301200 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:46.301244 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.301283 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:46.301359 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.309693 3010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35947
I20260619 16:35:46.309747 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35947 every 8 connection(s)
I20260619 16:35:46.310562 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:46.320636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3010
I20260619 16:35:46.320726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:46.321035 392 external_mini_cluster.cc:1428] Setting key d7c9417111dcea21a5aebf10d64d4efb
I20260619 16:35:46.323855 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:46.312216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56097 (local address 127.0.98.62:43529)
0619 16:35:46.312333 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:46.312337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:46.313083 (+ 746us) server_negotiation.cc:407] Connection header received
0619 16:35:46.314507 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:46.314518 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:46.314573 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:46.314671 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:46.315714 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.316240 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.317430 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.317651 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.317764 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:46.320118 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:46.320122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:46.320124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:46.320148 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:46.321788 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:46.322286 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:46.322290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:46.322292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:46.322382 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:46.322789 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:46.322792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:46.322793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:46.322952 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:46.323027 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:46.323651 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 16:35:46.323742 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:35:46.324760 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:46.312491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43529 (local address 127.0.98.2:56097)
0619 16:35:46.312942 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:46.312995 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:46.314308 (+ 1313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:46.314827 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:46.314834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:46.315155 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:46.315579 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:46.315591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.316361 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.316364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:46.317245 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:46.317253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.317795 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.317799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:46.317959 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:46.318512 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:46.318528 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:46.319972 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:46.321895 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:46.321899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:46.321909 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:46.322179 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:46.322479 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:46.322482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:46.322484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:46.322692 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:46.323057 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:46.323066 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:46.323392 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 16:35:46.323605 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 16:35:46.323942 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260619 16:35:46.325877 3139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43529
I20260619 16:35:46.326093 3139 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:46.326606 3139 heartbeater.cc:511] Master 127.0.98.62:43529 requested a full tablet report, sending...
I20260619 16:35:46.327589 2814 ts_manager.cc:194] Registered new tserver with Master: 079d5ed2a3804feaacad121e6cdd899f (127.0.98.2:35947)
I20260619 16:35:46.328125 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56097
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:46.349300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:46.455483 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:46.455739 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:46.455838 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:46.459988 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:46.460058 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:46.460168 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:46.465372 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:46.466295 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:46.467219 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:46.476650 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:46.476585 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:46.476590 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:46.476872 3144 server_base.cc:1034] running on GCE node
I20260619 16:35:46.477376 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:46.477988 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:46.479187 3144 hybrid_clock.cc:648] HybridClock initialized: now 1781886946479178 us; error 34 us; skew 500 ppm
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:46.481931 3144 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:46.482681 3144 webserver.cc:460] Webserver started at http://127.0.98.3:37949/ using document root <none> and password file <none>
I20260619 16:35:46.483162 3144 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:46.483217 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:46.483367 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:46.485380 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9d9ec16786c451b98dedfd1c2333f41"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "f23169a02081b6ba5196a3fc97d38395"
server_key_iv: "f4538c5630c1f60f776cad3fb1c5a107"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.485888 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9d9ec16786c451b98dedfd1c2333f41"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "f23169a02081b6ba5196a3fc97d38395"
server_key_iv: "f4538c5630c1f60f776cad3fb1c5a107"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.489415 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:46.491681 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.492866 3144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:46.492961 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f9d9ec16786c451b98dedfd1c2333f41"
format_stamp: "Formatted at 2026-06-19 16:35:46 on dist-test-slave-0f28"
server_key: "f23169a02081b6ba5196a3fc97d38395"
server_key_iv: "f4538c5630c1f60f776cad3fb1c5a107"
server_key_version: "encryptionkey@0"
I20260619 16:35:46.493041 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:46.511162 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:46.514261 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:46.514405 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:46.514900 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:46.515618 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:46.515671 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.515710 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:46.515726 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:46.524688 3144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34621
I20260619 16:35:46.524698 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34621 every 8 connection(s)
I20260619 16:35:46.525542 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:46.534663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3144
I20260619 16:35:46.534776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:46.535046 392 external_mini_cluster.cc:1428] Setting key d81b438a0aab9c907bbc89d6bdf9a9bf
I20260619 16:35:46.538743 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:46.527392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44171 (local address 127.0.98.62:43529)
0619 16:35:46.527526 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:46.527529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:46.528210 (+ 681us) server_negotiation.cc:407] Connection header received
0619 16:35:46.529605 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:46.529609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:46.529665 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:46.529763 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:46.530816 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.531343 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.532429 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.532686 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.532795 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:46.535189 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:46.535192 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:46.535194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:46.535218 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:46.536606 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:46.537115 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:46.537120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:46.537125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:46.537213 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:46.537575 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:46.537580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:46.537582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:46.537766 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:46.537874 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:46.538467 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 16:35:46.538581 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:35:46.539479 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:46.527543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43529 (local address 127.0.98.3:44171)
0619 16:35:46.528053 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:46.528092 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:46.529385 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:46.529917 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:46.529923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:46.530245 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:46.530678 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:46.530690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.531461 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.531465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:46.532308 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:46.532316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:46.532827 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:46.532831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:46.532992 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:46.533572 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:46.533587 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:46.535017 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:46.536743 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:46.536749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:46.536763 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:46.537011 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:46.537313 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:46.537339 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:46.537341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:46.537477 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:46.537880 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:46.537888 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:46.538199 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 16:35:46.538408 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 16:35:46.538725 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":125,"threads_started":1}
I20260619 16:35:46.540552 3273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43529
I20260619 16:35:46.540795 3273 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:46.541251 3273 heartbeater.cc:511] Master 127.0.98.62:43529 requested a full tablet report, sending...
I20260619 16:35:46.542239 2814 ts_manager.cc:194] Registered new tserver with Master: f9d9ec16786c451b98dedfd1c2333f41 (127.0.98.3:34621)
I20260619 16:35:46.542793 2814 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44171
I20260619 16:35:46.548612 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:35:46.561404 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:46.559620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.0.98.62:43529)
0619 16:35:46.559762 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:46.559767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:46.559955 (+ 188us) server_negotiation.cc:407] Connection header received
0619 16:35:46.560099 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:46.560102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:46.560147 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:46.560225 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:46.561205 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50174: BlockingRecv error: recv got EOF from 127.0.0.1:50174 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:35:46.562570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 2876
I20260619 16:35:46.568723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3010
I20260619 16:35:46.574457 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3144
I20260619 16:35:46.579434 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 2782
2026-06-19T16:35:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2083 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[3285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[3285](info): set up 2 sockets
Jun 19 16:35:46 dist-test-slave-0f28 krb5kdc[3285](info): commencing operation
krb5kdc: starting...
W20260619 16:35:47.649260 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:47 dist-test-slave-0f28 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:47.768491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:47.873210 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:47.873466 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:47.873575 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:47.878153 3298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:47.878206 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:47.878225 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:47.878243 3298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:47.878258 3298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:47.883280 3298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:47.884234 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:47.885178 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:47.893186 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:47.893276 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:47.893257 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:47.893440 3298 server_base.cc:1034] running on GCE node
I20260619 16:35:47.894038 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:47.895095 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:47.896277 3298 hybrid_clock.cc:648] HybridClock initialized: now 1781886947896245 us; error 47 us; skew 500 ppm
Jun 19 16:35:47 dist-test-slave-0f28 krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:47.898957 3298 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:47.899772 3298 webserver.cc:460] Webserver started at http://127.0.98.62:36807/ using document root <none> and password file <none>
I20260619 16:35:47.900306 3298 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:47.900396 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:47.900614 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:47.902586 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "4999c22b3bca4d6f85b065e320f9cec9"
format_stamp: "Formatted at 2026-06-19 16:35:47 on dist-test-slave-0f28"
server_key: "cf5977190f66a2e43c4346aa7d2f9173"
server_key_iv: "491033ae5eec6e384e96ec3fa5678802"
server_key_version: "encryptionkey@0"
I20260619 16:35:47.903123 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "4999c22b3bca4d6f85b065e320f9cec9"
format_stamp: "Formatted at 2026-06-19 16:35:47 on dist-test-slave-0f28"
server_key: "cf5977190f66a2e43c4346aa7d2f9173"
server_key_iv: "491033ae5eec6e384e96ec3fa5678802"
server_key_version: "encryptionkey@0"
I20260619 16:35:47.906579 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:47.908771 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:47.909919 3298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:47.910059 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "4999c22b3bca4d6f85b065e320f9cec9"
format_stamp: "Formatted at 2026-06-19 16:35:47 on dist-test-slave-0f28"
server_key: "cf5977190f66a2e43c4346aa7d2f9173"
server_key_iv: "491033ae5eec6e384e96ec3fa5678802"
server_key_version: "encryptionkey@0"
I20260619 16:35:47.910175 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:47.934635 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:47.937467 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:47.937662 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:47.944229 3298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39281
I20260619 16:35:47.944492 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39281 every 8 connection(s)
I20260619 16:35:47.945164 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:47.947750 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:47.953045 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9: Bootstrap starting.
I20260619 16:35:47.954149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3298
I20260619 16:35:47.954270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:47.954548 392 external_mini_cluster.cc:1428] Setting key e5735d33254c88ce16696c805705bb59
I20260619 16:35:47.955899 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:47.956630 3366 log.cc:824] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:47.958765 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9: No bootstrap required, opened a new log
I20260619 16:35:47.963187 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4999c22b3bca4d6f85b065e320f9cec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39281 } }
I20260619 16:35:47.963389 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Jun 19 16:35:47 dist-test-slave-0f28 krb5kdc[3285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:47.963456 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4999c22b3bca4d6f85b065e320f9cec9, State: Initialized, Role: FOLLOWER
I20260619 16:35:47.963892 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4999c22b3bca4d6f85b065e320f9cec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39281 } }
I20260619 16:35:47.964067 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:47.964141 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:47.964246 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:47.965435 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4999c22b3bca4d6f85b065e320f9cec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39281 } }
I20260619 16:35:47.965790 3366 leader_election.cc:302] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4999c22b3bca4d6f85b065e320f9cec9; no voters:
I20260619 16:35:47.966161 3366 leader_election.cc:288] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:47.966254 3371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:47.966470 3371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [term 1 LEADER]: Becoming Leader. State: Replica: 4999c22b3bca4d6f85b065e320f9cec9, State: Running, Role: LEADER
I20260619 16:35:47.966849 3371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4999c22b3bca4d6f85b065e320f9cec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39281 } }
I20260619 16:35:47.967443 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:47.968336 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4999c22b3bca4d6f85b065e320f9cec9. Latest consensus state: current_term: 1 leader_uuid: "4999c22b3bca4d6f85b065e320f9cec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4999c22b3bca4d6f85b065e320f9cec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39281 } } }
I20260619 16:35:47.968456 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:47.968843 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4999c22b3bca4d6f85b065e320f9cec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4999c22b3bca4d6f85b065e320f9cec9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39281 } } }
I20260619 16:35:47.968976 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:47.969499 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:47.955484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.0.98.62:39281)
0619 16:35:47.955899 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:47.955913 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:47.955967 (+ 54us) server_negotiation.cc:407] Connection header received
0619 16:35:47.956594 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:47.956613 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:47.957010 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:47.957420 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:47.958368 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:47.959417 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:47.960636 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:47.961089 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:47.961221 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:47.964017 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:47.964034 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:47.964049 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:47.964082 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:47.965980 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:47.966514 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:47.966521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:47.966530 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:47.966665 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:47.966991 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:47.966994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:47.966996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:47.967472 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:47.967639 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:47.968117 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 16:35:47.968448 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":119,"threads_started":1}
I20260619 16:35:47.970136 3380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:47.972558 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:35:47.972849 3380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:47.977362 3380 catalog_manager.cc:1347] Generated new cluster ID: 7bf713e61d724b3cbc156112bc2540e4
I20260619 16:35:47.977424 3380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:47.990435 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:47.985962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56194 (local address 127.0.98.62:39281)
0619 16:35:47.986076 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:47.986081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:47.986254 (+ 173us) server_negotiation.cc:407] Connection header received
0619 16:35:47.986399 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:47.986402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:47.986450 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:47.986532 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:47.987034 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:47.987527 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:47.988364 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:47.988605 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:47.988724 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:47.989580 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:47.989589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:47.989591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:47.990008 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:47.990036 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:47.990051 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:47.990150 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:47.990240 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 16:35:47.990294 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:35:47.996068 3380 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:47.996893 3380 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:48.000519 3380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4999c22b3bca4d6f85b065e320f9cec9: Generated new TSK 0
I20260619 16:35:48.001219 3380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:35:48.009739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3298
2026-06-19T16:35:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1427 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:48 dist-test-slave-0f28 krb5kdc[3399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:48 dist-test-slave-0f28 krb5kdc[3399](info): set up 2 sockets
Jun 19 16:35:48 dist-test-slave-0f28 krb5kdc[3399](info): commencing operation
krb5kdc: starting...
W20260619 16:35:49.061645 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:49.180066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39505
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:49.285557 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:49.285782 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:49.285818 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:49.289712 3412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:49.289757 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:49.289776 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:49.289793 3412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:49.289808 3412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:49.294867 3412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:49.295778 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:49.296686 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:49.304656 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:49.304670 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:49.304791 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:49.304791 3412 server_base.cc:1034] running on GCE node
I20260619 16:35:49.305490 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:49.306478 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:49.307652 3412 hybrid_clock.cc:648] HybridClock initialized: now 1781886949307626 us; error 42 us; skew 500 ppm
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:49.310233 3412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:49.311023 3412 webserver.cc:460] Webserver started at http://127.0.98.62:46117/ using document root <none> and password file <none>
I20260619 16:35:49.311517 3412 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:49.311617 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:49.311825 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:49.313709 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "03123a19eb464e8db4544bbc706a3e62"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "44d3cfd7c10a9ec4ac8ce544ea905655"
server_key_iv: "1cfe2064a26d9e19f47c8c9f922dfcfa"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.314226 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "03123a19eb464e8db4544bbc706a3e62"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "44d3cfd7c10a9ec4ac8ce544ea905655"
server_key_iv: "1cfe2064a26d9e19f47c8c9f922dfcfa"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.317540 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:49.319739 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.320822 3412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:49.320948 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "03123a19eb464e8db4544bbc706a3e62"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "44d3cfd7c10a9ec4ac8ce544ea905655"
server_key_iv: "1cfe2064a26d9e19f47c8c9f922dfcfa"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.321050 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:49.335449 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:49.338246 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:49.338431 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:49.345232 3412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39505
I20260619 16:35:49.345520 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39505 every 8 connection(s)
I20260619 16:35:49.346210 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:49.348762 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:49.353869 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62: Bootstrap starting.
I20260619 16:35:49.355309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3412
I20260619 16:35:49.355398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:49.355683 392 external_mini_cluster.cc:1428] Setting key 6ef9e5fdeb20b4ee86a6cf6ec0ba7c7f
I20260619 16:35:49.356300 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:49.357036 3480 log.cc:824] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:49.358877 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62: No bootstrap required, opened a new log
I20260619 16:35:49.362807 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03123a19eb464e8db4544bbc706a3e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39505 } }
I20260619 16:35:49.363077 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:49.363143 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03123a19eb464e8db4544bbc706a3e62, State: Initialized, Role: FOLLOWER
I20260619 16:35:49.363641 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03123a19eb464e8db4544bbc706a3e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39505 } }
I20260619 16:35:49.363786 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:49.363868 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:49.363987 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:49.364902 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03123a19eb464e8db4544bbc706a3e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39505 } }
I20260619 16:35:49.365324 3480 leader_election.cc:302] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03123a19eb464e8db4544bbc706a3e62; no voters:
I20260619 16:35:49.365698 3480 leader_election.cc:288] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:49.365801 3485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:49.366011 3485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [term 1 LEADER]: Becoming Leader. State: Replica: 03123a19eb464e8db4544bbc706a3e62, State: Running, Role: LEADER
I20260619 16:35:49.366333 3485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03123a19eb464e8db4544bbc706a3e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39505 } }
I20260619 16:35:49.366847 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:49.367846 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03123a19eb464e8db4544bbc706a3e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03123a19eb464e8db4544bbc706a3e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39505 } } }
I20260619 16:35:49.367961 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:49.368251 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03123a19eb464e8db4544bbc706a3e62. Latest consensus state: current_term: 1 leader_uuid: "03123a19eb464e8db4544bbc706a3e62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03123a19eb464e8db4544bbc706a3e62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39505 } } }
I20260619 16:35:49.368392 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:49.368662 3494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:49.370374 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:49.356723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.0.98.62:39505)
0619 16:35:49.357158 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:49.357177 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:49.357216 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:35:49.358121 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:49.358136 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:49.358430 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:49.358697 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:49.359567 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.360508 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.361764 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.362206 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.362316 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:49.365001 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:49.365006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:49.365017 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:49.365071 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:49.367076 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:49.367594 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:49.367602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:49.367612 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:49.367765 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:49.368142 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:49.368146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:49.368148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:49.368649 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:49.368840 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:49.369405 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 16:35:49.369720 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260619 16:35:49.372191 3494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:49.377219 3494 catalog_manager.cc:1347] Generated new cluster ID: da80d3ad1e884437b81ee348102c4607
I20260619 16:35:49.377279 3494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:49.391028 3494 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:49.391875 3494 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:49.395308 3494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03123a19eb464e8db4544bbc706a3e62: Generated new TSK 0
I20260619 16:35:49.395896 3494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:49.425889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39505
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:49.551155 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:49.551443 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:49.551523 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:49.555951 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:49.556041 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:49.556135 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:49.561831 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:49.562918 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:49.563961 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:35:49.573949 3506 server_base.cc:1034] running on GCE node
W20260619 16:35:49.573961 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:49.573987 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:49.573931 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:49.574661 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:49.575403 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:49.576624 3506 hybrid_clock.cc:648] HybridClock initialized: now 1781886949576591 us; error 48 us; skew 500 ppm
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:49.585877 3506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:49.586813 3506 webserver.cc:460] Webserver started at http://127.0.98.1:39855/ using document root <none> and password file <none>
I20260619 16:35:49.587404 3506 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:49.587468 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:49.587690 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:49.589790 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc8fb99d10714513a50e165a12ad0b19"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "edfa515b41a015f0bf4fbf73ce7d8445"
server_key_iv: "f113c022f3bdb8a88e23525d20ea9947"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.590350 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc8fb99d10714513a50e165a12ad0b19"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "edfa515b41a015f0bf4fbf73ce7d8445"
server_key_iv: "f113c022f3bdb8a88e23525d20ea9947"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.594187 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:35:49.596791 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.598153 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:49.598258 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "fc8fb99d10714513a50e165a12ad0b19"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "edfa515b41a015f0bf4fbf73ce7d8445"
server_key_iv: "f113c022f3bdb8a88e23525d20ea9947"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.598341 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:49.618027 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:49.621248 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:49.621461 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:49.621980 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:49.623055 3506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:49.623114 3506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.623167 3506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:49.623195 3506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.631781 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37089
I20260619 16:35:49.631847 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37089 every 8 connection(s)
I20260619 16:35:49.632725 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:49.642457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3506
I20260619 16:35:49.642587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:49.642863 392 external_mini_cluster.cc:1428] Setting key c7d07b716b8a3fda95659559e457ae6f
I20260619 16:35:49.646293 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:49.634516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39179 (local address 127.0.98.62:39505)
0619 16:35:49.634680 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:49.634685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:49.635538 (+ 853us) server_negotiation.cc:407] Connection header received
0619 16:35:49.636974 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:49.636977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:49.637023 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:49.637101 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:49.638245 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.638803 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.639912 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.640153 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.640241 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:49.642549 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:49.642554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:49.642564 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:49.642596 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:49.644159 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:49.644672 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:49.644676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:49.644678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:49.644774 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:49.645170 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:49.645177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:49.645182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:49.645461 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:49.645578 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:49.645993 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 16:35:49.646136 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:35:49.647377 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:49.634904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39505 (local address 127.0.98.1:39179)
0619 16:35:49.635370 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:49.635421 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:49.636770 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:49.637254 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:49.637261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:49.637645 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:49.638104 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:49.638117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.638919 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.638922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:49.639792 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:49.639800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.640286 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.640289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:49.640424 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:49.640919 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:49.640934 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:49.642389 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:49.644269 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:49.644274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:49.644286 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:49.644570 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:49.644889 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:49.644894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:49.644897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:49.645078 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:49.645567 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:49.645571 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:49.645771 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 16:35:49.645916 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 16:35:49.646273 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":120,"threads_started":1}
I20260619 16:35:49.648543 3635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39505
I20260619 16:35:49.648803 3635 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:49.649258 3635 heartbeater.cc:511] Master 127.0.98.62:39505 requested a full tablet report, sending...
I20260619 16:35:49.650738 3444 ts_manager.cc:194] Registered new tserver with Master: fc8fb99d10714513a50e165a12ad0b19 (127.0.98.1:37089)
I20260619 16:35:49.651811 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39179
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:49.669085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39505
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:49.774637 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:49.774891 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:49.775000 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:49.778949 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:49.779024 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:49.779138 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:49.784215 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:49.785142 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:49.786127 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:49.795305 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:49.795360 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:49.795560 3640 server_base.cc:1034] running on GCE node
W20260619 16:35:49.795311 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:49.795980 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:49.796607 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:49.797784 3640 hybrid_clock.cc:648] HybridClock initialized: now 1781886949797771 us; error 25 us; skew 500 ppm
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:49.800479 3640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:49.801292 3640 webserver.cc:460] Webserver started at http://127.0.98.2:40499/ using document root <none> and password file <none>
I20260619 16:35:49.801877 3640 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:49.801964 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:49.802217 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:49.804111 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "c570fb9a7f934c51b6e12c1b52811604"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "66e87bf405c8d103e0e860f904329370"
server_key_iv: "d4b8a7baf880f64bd3592622d45d353a"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.804652 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c570fb9a7f934c51b6e12c1b52811604"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "66e87bf405c8d103e0e860f904329370"
server_key_iv: "d4b8a7baf880f64bd3592622d45d353a"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.808255 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:35:49.810629 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.811795 3640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:35:49.811928 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "c570fb9a7f934c51b6e12c1b52811604"
format_stamp: "Formatted at 2026-06-19 16:35:49 on dist-test-slave-0f28"
server_key: "66e87bf405c8d103e0e860f904329370"
server_key_iv: "d4b8a7baf880f64bd3592622d45d353a"
server_key_version: "encryptionkey@0"
I20260619 16:35:49.812036 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:49.828495 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:49.831178 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:49.831368 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:49.831898 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:49.832655 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:49.832726 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.832792 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:49.832841 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:49.841486 3640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36417
I20260619 16:35:49.841513 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36417 every 8 connection(s)
I20260619 16:35:49.842298 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:49.844290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3640
I20260619 16:35:49.844436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:49.844817 392 external_mini_cluster.cc:1428] Setting key 4cc251de2fe2fb29cac24ad32e18b95a
Jun 19 16:35:49 dist-test-slave-0f28 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:35:49.857448 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:49.843940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54197 (local address 127.0.98.62:39505)
0619 16:35:49.844078 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:49.844082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:49.845051 (+ 969us) server_negotiation.cc:407] Connection header received
0619 16:35:49.846803 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:49.846805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:49.846848 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:49.846921 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:49.848360 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.848883 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.850717 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.850933 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.851029 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:49.853838 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:49.853841 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:49.853843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:49.853865 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:49.855294 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:49.855840 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:49.855850 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:49.855852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:49.855919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:49.856303 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:49.856306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:49.856307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:49.856473 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:49.856589 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:49.857139 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 16:35:49.857249 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:35:49.858312 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:49.844302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39505 (local address 127.0.98.2:54197)
0619 16:35:49.844870 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:49.844926 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:49.846566 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:49.847100 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:49.847111 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:49.847613 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:49.848221 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:49.848236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.849015 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.849022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:49.850582 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:49.850597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:49.851099 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:49.851106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:49.851334 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:49.851926 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:49.851950 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:49.853671 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:49.855409 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:49.855415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:49.855430 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:49.855733 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:49.856022 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:49.856025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:49.856027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:49.856156 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:49.856601 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:49.856606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:49.856883 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 16:35:49.857081 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 16:35:49.857405 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":191,"threads_started":1}
I20260619 16:35:49.859391 3769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39505
I20260619 16:35:49.859624 3769 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:49.860163 3769 heartbeater.cc:511] Master 127.0.98.62:39505 requested a full tablet report, sending...
I20260619 16:35:49.861243 3444 ts_manager.cc:194] Registered new tserver with Master: c570fb9a7f934c51b6e12c1b52811604 (127.0.98.2:36417)
I20260619 16:35:49.861893 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:49.872435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39505
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:49.978945 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:49.979206 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:49.979249 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:49.983443 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:49.983500 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:49.983592 3774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:49.988737 3774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:49.989794 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:49.990787 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:50.000269 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:50.000168 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:50.000164 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:50.000232 3774 server_base.cc:1034] running on GCE node
I20260619 16:35:50.000902 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:50.001677 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:50.002866 3774 hybrid_clock.cc:648] HybridClock initialized: now 1781886950002852 us; error 37 us; skew 500 ppm
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:50.005570 3774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:50.006361 3774 webserver.cc:460] Webserver started at http://127.0.98.3:45063/ using document root <none> and password file <none>
I20260619 16:35:50.006915 3774 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:50.006978 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:50.007141 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:50.009110 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "8af0d25548504c28befeec8692859773"
format_stamp: "Formatted at 2026-06-19 16:35:50 on dist-test-slave-0f28"
server_key: "c673192095e84d71de55a4a7f184ed6b"
server_key_iv: "31b19fbe28709df3000ed26d40e30c7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:50.009680 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8af0d25548504c28befeec8692859773"
format_stamp: "Formatted at 2026-06-19 16:35:50 on dist-test-slave-0f28"
server_key: "c673192095e84d71de55a4a7f184ed6b"
server_key_iv: "31b19fbe28709df3000ed26d40e30c7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:50.013172 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:50.015507 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:50.016723 3774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:50.016836 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "8af0d25548504c28befeec8692859773"
format_stamp: "Formatted at 2026-06-19 16:35:50 on dist-test-slave-0f28"
server_key: "c673192095e84d71de55a4a7f184ed6b"
server_key_iv: "31b19fbe28709df3000ed26d40e30c7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:50.016933 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:50.036506 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:50.039369 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:50.039561 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:50.040102 3774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:50.040865 3774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:50.040908 3774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:50.040968 3774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:50.041004 3774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:50.049836 3774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34737
I20260619 16:35:50.049930 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34737 every 8 connection(s)
I20260619 16:35:50.050678 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:50.058166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3774
I20260619 16:35:50.058315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:50.058708 392 external_mini_cluster.cc:1428] Setting key ec59330abfc2675bf47f8e8ddbaec741
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:50.063889 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:50.052421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57225 (local address 127.0.98.62:39505)
0619 16:35:50.052558 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:50.052561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:50.053351 (+ 790us) server_negotiation.cc:407] Connection header received
0619 16:35:50.054724 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:50.054727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:50.054781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:50.054886 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:50.055949 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:50.056469 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:50.057648 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:50.057866 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:50.057976 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:50.060669 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:50.060674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:50.060677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:50.060728 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:50.062046 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:50.062517 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:50.062520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:50.062521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:50.062588 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:50.062913 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:50.062916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:50.062918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:50.063102 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:50.063192 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:50.063671 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 16:35:50.063775 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:35:50.064613 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:50.052763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39505 (local address 127.0.98.3:57225)
0619 16:35:50.053190 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:50.053231 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:50.054523 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:50.055045 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:50.055052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:50.055382 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:50.055815 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:50.055826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:50.056590 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:50.056593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:50.057524 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:50.057534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:50.058009 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:50.058013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:50.058182 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:50.058899 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:50.058920 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:50.060513 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:50.062161 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:50.062165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:50.062175 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:50.062430 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:50.062665 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:50.062667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:50.062668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:50.062789 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:50.063193 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:50.063198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:50.063418 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 16:35:50.063567 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:35:50.063837 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260619 16:35:50.065693 3903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39505
I20260619 16:35:50.065910 3903 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:50.066354 3903 heartbeater.cc:511] Master 127.0.98.62:39505 requested a full tablet report, sending...
I20260619 16:35:50.067387 3444 ts_manager.cc:194] Registered new tserver with Master: 8af0d25548504c28befeec8692859773 (127.0.98.3:34737)
I20260619 16:35:50.067960 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57225
I20260619 16:35:50.072453 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886950, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:50.096963 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:50.088563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.98.62:39505)
0619 16:35:50.088758 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:50.088763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:50.088874 (+ 111us) server_negotiation.cc:407] Connection header received
0619 16:35:50.089052 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:50.089055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:50.089098 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:50.089187 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:50.089950 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:50.090457 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:50.091294 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:50.091508 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:50.091619 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:50.093946 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:50.093949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:50.093951 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:50.093972 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:50.095289 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:50.095694 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:50.095696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:50.095697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:50.095766 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:50.096070 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:50.096073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:50.096074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:50.096399 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:50.096535 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:50.096744 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:35:50.096844 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 16:35:50.097431 3444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51048
I20260619 16:35:50.098379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3506
I20260619 16:35:50.103766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3640
I20260619 16:35:50.109225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3774
I20260619 16:35:50.114391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3412
2026-06-19T16:35:50Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2107 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3915](info): set up 2 sockets
Jun 19 16:35:50 dist-test-slave-0f28 krb5kdc[3915](info): commencing operation
krb5kdc: starting...
W20260619 16:35:51.157395 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:51.280802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:51.386023 3928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:51.386273 3928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:51.386312 3928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:51.390409 3928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:51.390460 3928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:51.390478 3928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:51.390494 3928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:51.390537 3928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:51.395594 3928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:51.396543 3928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:51.397514 3928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:51.405561 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:51.405566 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:51.405695 3928 server_base.cc:1034] running on GCE node
W20260619 16:35:51.405570 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:51.406275 3928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:51.407236 3928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:51.408408 3928 hybrid_clock.cc:648] HybridClock initialized: now 1781886951408388 us; error 29 us; skew 500 ppm
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:51.411088 3928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:51.411904 3928 webserver.cc:460] Webserver started at http://127.0.98.62:44407/ using document root <none> and password file <none>
I20260619 16:35:51.412386 3928 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:51.412439 3928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:51.412591 3928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:51.414503 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "1235a09d33ef48018c4f1963bc7045ce"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "ed3ea49dc271e5d350772c35a6804917"
server_key_iv: "a3a96b5d846bc43664b3b9fefa20ad63"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.414976 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "1235a09d33ef48018c4f1963bc7045ce"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "ed3ea49dc271e5d350772c35a6804917"
server_key_iv: "a3a96b5d846bc43664b3b9fefa20ad63"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.418526 3928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:51.420668 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.421923 3928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:51.422056 3928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "1235a09d33ef48018c4f1963bc7045ce"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "ed3ea49dc271e5d350772c35a6804917"
server_key_iv: "a3a96b5d846bc43664b3b9fefa20ad63"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.422170 3928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:51.446128 3928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:51.449182 3928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:51.449394 3928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:51.456249 3928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38801
I20260619 16:35:51.456274 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38801 every 8 connection(s)
I20260619 16:35:51.457129 3928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:51.459728 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:51.464860 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce: Bootstrap starting.
I20260619 16:35:51.466136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 3928
I20260619 16:35:51.466285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:51.466547 392 external_mini_cluster.cc:1428] Setting key c7148eb7e85bcff97a5d061f8caa633d
I20260619 16:35:51.467365 3996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:51.468173 3996 log.cc:824] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:51.470176 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce: No bootstrap required, opened a new log
I20260619 16:35:51.474269 3996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1235a09d33ef48018c4f1963bc7045ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 16:35:51.474532 3996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:51.474587 3996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1235a09d33ef48018c4f1963bc7045ce, State: Initialized, Role: FOLLOWER
I20260619 16:35:51.475019 3996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1235a09d33ef48018c4f1963bc7045ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 16:35:51.475167 3996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:51.475255 3996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:51.475350 3996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:51.476310 3996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1235a09d33ef48018c4f1963bc7045ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 16:35:51.476679 3996 leader_election.cc:302] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1235a09d33ef48018c4f1963bc7045ce; no voters:
I20260619 16:35:51.477038 3996 leader_election.cc:288] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:51.477136 4001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:51.477386 4001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [term 1 LEADER]: Becoming Leader. State: Replica: 1235a09d33ef48018c4f1963bc7045ce, State: Running, Role: LEADER
I20260619 16:35:51.477705 4001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1235a09d33ef48018c4f1963bc7045ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } }
I20260619 16:35:51.478281 3996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:51.479110 4002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1235a09d33ef48018c4f1963bc7045ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1235a09d33ef48018c4f1963bc7045ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } } }
I20260619 16:35:51.479233 4002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:51.479611 4010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:51.479671 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1235a09d33ef48018c4f1963bc7045ce. Latest consensus state: current_term: 1 leader_uuid: "1235a09d33ef48018c4f1963bc7045ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1235a09d33ef48018c4f1963bc7045ce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38801 } } }
I20260619 16:35:51.479820 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:51.481824 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:51.467602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.0.98.62:38801)
0619 16:35:51.467996 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:51.468010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:51.468043 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:35:51.468686 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:51.468704 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:51.469000 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:51.469282 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:51.470158 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.471162 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.472324 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.472793 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.472907 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:51.475933 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:51.475938 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:51.475950 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:51.475973 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:51.477951 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:51.478449 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:51.478455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:51.478461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:51.478604 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:51.478918 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:51.478922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:51.478924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:51.479455 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:51.480363 (+ 908us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:51.480549 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:35:51.480879 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
I20260619 16:35:51.482699 4010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:51.488139 4010 catalog_manager.cc:1347] Generated new cluster ID: b045515c8e5f4622863e39ad220edfe2
I20260619 16:35:51.488217 4010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:51.497859 4010 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:51.498692 4010 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:51.515030 4010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1235a09d33ef48018c4f1963bc7045ce: Generated new TSK 0
I20260619 16:35:51.515695 4010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:51.550122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:51.657125 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:51.657403 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:51.657449 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:51.661291 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:51.661357 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:51.661412 4022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:51.666862 4022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:51.667817 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:51.668776 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:51.678238 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:51.678332 4022 server_base.cc:1034] running on GCE node
W20260619 16:35:51.678238 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:51.678319 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:51.678934 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:51.679497 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:51.680673 4022 hybrid_clock.cc:648] HybridClock initialized: now 1781886951680655 us; error 32 us; skew 500 ppm
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:51.683466 4022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:51.684301 4022 webserver.cc:460] Webserver started at http://127.0.98.1:35513/ using document root <none> and password file <none>
I20260619 16:35:51.684857 4022 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:51.684950 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:51.685176 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:51.687244 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "c0be4f53c8a64b218e523342d082fa65"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "d44abf8bf57bebb14ceedd35e4a6b95a"
server_key_iv: "9ca4623ecbae81593e7405ff58ea1f6b"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.687808 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c0be4f53c8a64b218e523342d082fa65"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "d44abf8bf57bebb14ceedd35e4a6b95a"
server_key_iv: "9ca4623ecbae81593e7405ff58ea1f6b"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.691462 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:51.693966 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.695192 4022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:51.695336 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "c0be4f53c8a64b218e523342d082fa65"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "d44abf8bf57bebb14ceedd35e4a6b95a"
server_key_iv: "9ca4623ecbae81593e7405ff58ea1f6b"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.695451 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:51.708092 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:51.711512 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:51.711725 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:51.712301 4022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:51.713115 4022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:51.713186 4022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.713258 4022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:51.713299 4022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.722347 4022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43377
I20260619 16:35:51.722393 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43377 every 8 connection(s)
I20260619 16:35:51.723204 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:51.725658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4022
I20260619 16:35:51.725786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:51.726181 392 external_mini_cluster.cc:1428] Setting key fe6095a1df51c19b66c4f71fce8c9370
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:35:51.737470 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:51.724979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50891 (local address 127.0.98.62:38801)
0619 16:35:51.725112 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:51.725117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:51.725987 (+ 870us) server_negotiation.cc:407] Connection header received
0619 16:35:51.727726 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:51.727730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:51.727781 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:51.727858 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:51.728969 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.729591 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.730732 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.730972 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.731083 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:51.733780 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:51.733784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:51.733787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:51.733817 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:51.735319 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:51.735806 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:51.735808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:51.735810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:51.735884 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:51.736226 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:51.736228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:51.736229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:51.736447 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:51.736551 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:51.737059 (+ 508us) server_negotiation.cc:299] Negotiation successful
0619 16:35:51.737192 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:35:51.738659 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:51.725304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38801 (local address 127.0.98.1:50891)
0619 16:35:51.725804 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:51.725855 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:51.727529 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:51.728026 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:51.728033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:51.728366 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:51.728821 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:51.728833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.729717 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.729720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:51.730605 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:51.730614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.731118 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.731123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:51.731296 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:51.731902 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:51.731918 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:51.733614 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:51.735439 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:51.735444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:51.735457 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:51.735704 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:51.736007 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:51.736009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:51.736011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:51.736143 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:51.736564 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:51.736570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:51.736806 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 16:35:51.737028 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 16:35:51.737472 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260619 16:35:51.740033 4151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38801
I20260619 16:35:51.740301 4151 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:51.740796 4151 heartbeater.cc:511] Master 127.0.98.62:38801 requested a full tablet report, sending...
I20260619 16:35:51.742524 3960 ts_manager.cc:194] Registered new tserver with Master: c0be4f53c8a64b218e523342d082fa65 (127.0.98.1:43377)
I20260619 16:35:51.744163 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:51.756273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:51.866868 4156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:51.867115 4156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:51.867154 4156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:51.871083 4156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:51.871155 4156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:51.871222 4156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:51.876269 4156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:51.877180 4156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:51.878171 4156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:51.887630 4162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:51.887719 4156 server_base.cc:1034] running on GCE node
W20260619 16:35:51.887669 4164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:51.887652 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:51.888254 4156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:51.888857 4156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:51.890043 4156 hybrid_clock.cc:648] HybridClock initialized: now 1781886951890021 us; error 28 us; skew 500 ppm
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:51.892907 4156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:51.893850 4156 webserver.cc:460] Webserver started at http://127.0.98.2:44721/ using document root <none> and password file <none>
I20260619 16:35:51.894411 4156 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:51.894502 4156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:51.894727 4156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:51.896772 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "24834697e69a4a57931a69812fe9ede0"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "108945387579fbe56a472540fb44c081"
server_key_iv: "78452e0f3a47e77c750f628dea2d715a"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.897384 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24834697e69a4a57931a69812fe9ede0"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "108945387579fbe56a472540fb44c081"
server_key_iv: "78452e0f3a47e77c750f628dea2d715a"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.901494 4156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:35:51.904063 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.905645 4156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:51.905799 4156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "24834697e69a4a57931a69812fe9ede0"
format_stamp: "Formatted at 2026-06-19 16:35:51 on dist-test-slave-0f28"
server_key: "108945387579fbe56a472540fb44c081"
server_key_iv: "78452e0f3a47e77c750f628dea2d715a"
server_key_version: "encryptionkey@0"
I20260619 16:35:51.905921 4156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:51.922758 4156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:51.925700 4156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:51.925890 4156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:51.926425 4156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:51.927202 4156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:51.927246 4156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.927313 4156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:51.927366 4156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:51.935648 4156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43225
I20260619 16:35:51.935734 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43225 every 8 connection(s)
I20260619 16:35:51.936570 4156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:51.942605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4156
I20260619 16:35:51.942731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:51.943006 392 external_mini_cluster.cc:1428] Setting key 3aa36f125f53d1cf406d0f6ad16eeaab
Jun 19 16:35:51 dist-test-slave-0f28 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:51.950713 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:51.938369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38451 (local address 127.0.98.62:38801)
0619 16:35:51.938497 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:51.938508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:51.939244 (+ 736us) server_negotiation.cc:407] Connection header received
0619 16:35:51.940645 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:51.940649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:51.940709 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:51.940814 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:51.942010 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.942654 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.943969 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.944236 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.944319 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:51.946820 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:51.946824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:51.946826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:51.946848 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:51.948496 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:51.948967 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:51.948971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:51.948972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:51.949079 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:51.949452 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:51.949458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:51.949461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:51.949726 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:51.949838 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:51.950357 (+ 519us) server_negotiation.cc:299] Negotiation successful
0619 16:35:51.950494 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:35:51.951301 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:51.938650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38801 (local address 127.0.98.2:38451)
0619 16:35:51.939091 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:51.939131 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:51.940416 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:51.940983 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:51.940989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:51.941409 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:51.941862 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:51.941874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.942828 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.942832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:51.943858 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:51.943867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:51.944361 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:51.944368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:51.944519 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:51.945094 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:51.945109 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:51.946656 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:51.948613 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:51.948617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:51.948627 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:51.948868 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:51.949177 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:51.949179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:51.949180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:51.949301 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:51.949863 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:51.949875 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:51.950085 (+ 210us) client_negotiation.cc:769] Sending connection context
0619 16:35:51.950239 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 16:35:51.950463 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
I20260619 16:35:51.952379 4285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38801
I20260619 16:35:51.952643 4285 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:51.953182 4285 heartbeater.cc:511] Master 127.0.98.62:38801 requested a full tablet report, sending...
I20260619 16:35:51.954324 3960 ts_manager.cc:194] Registered new tserver with Master: 24834697e69a4a57931a69812fe9ede0 (127.0.98.2:43225)
I20260619 16:35:51.954941 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:51.970036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:52.074884 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:52.075119 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:52.075160 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:52.079352 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:52.079404 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:52.079485 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:52.084502 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:52.085461 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:52.086395 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:52.095542 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:52.095646 4290 server_base.cc:1034] running on GCE node
W20260619 16:35:52.095577 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:52.095583 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:52.096156 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:52.096784 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:52.097958 4290 hybrid_clock.cc:648] HybridClock initialized: now 1781886952097950 us; error 30 us; skew 500 ppm
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:52.100683 4290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:52.101532 4290 webserver.cc:460] Webserver started at http://127.0.98.3:40761/ using document root <none> and password file <none>
I20260619 16:35:52.102056 4290 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:52.102144 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:52.102356 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:52.104267 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "35afca84e2e54bfa8ccc807109ad418a"
format_stamp: "Formatted at 2026-06-19 16:35:52 on dist-test-slave-0f28"
server_key: "4fb0ca57ee9cc184c9b3fca82537eb42"
server_key_iv: "0e80a09b6b331d2a92298f8c0ea6212f"
server_key_version: "encryptionkey@0"
I20260619 16:35:52.104812 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "35afca84e2e54bfa8ccc807109ad418a"
format_stamp: "Formatted at 2026-06-19 16:35:52 on dist-test-slave-0f28"
server_key: "4fb0ca57ee9cc184c9b3fca82537eb42"
server_key_iv: "0e80a09b6b331d2a92298f8c0ea6212f"
server_key_version: "encryptionkey@0"
I20260619 16:35:52.108408 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:35:52.111065 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:52.112342 4290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 16:35:52.112489 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "35afca84e2e54bfa8ccc807109ad418a"
format_stamp: "Formatted at 2026-06-19 16:35:52 on dist-test-slave-0f28"
server_key: "4fb0ca57ee9cc184c9b3fca82537eb42"
server_key_iv: "0e80a09b6b331d2a92298f8c0ea6212f"
server_key_version: "encryptionkey@0"
I20260619 16:35:52.112602 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:52.141198 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:52.144599 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:52.144809 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:52.145418 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:52.146270 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:52.146339 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:52.146410 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:52.146466 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:52.156071 4290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44751
I20260619 16:35:52.156095 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44751 every 8 connection(s)
I20260619 16:35:52.156927 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:52.165575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4290
I20260619 16:35:52.165704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:52.165992 392 external_mini_cluster.cc:1428] Setting key 659ae07dc4b6ebaee399d6820f1dc168
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:52.170433 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:52.158757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36293 (local address 127.0.98.62:38801)
0619 16:35:52.158917 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:52.158922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:52.159729 (+ 807us) server_negotiation.cc:407] Connection header received
0619 16:35:52.161002 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:52.161004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:52.161048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:52.161121 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:52.162301 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.162828 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.163930 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.164138 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.164228 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:52.166831 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:52.166837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:52.166841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:52.166868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:52.168409 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:52.168926 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:52.168929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:52.168930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:52.169006 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:52.169451 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:52.169454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:52.169455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:52.169655 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:52.169743 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:52.170223 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 16:35:52.170319 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:35:52.171228 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:52.159128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38801 (local address 127.0.98.3:36293)
0619 16:35:52.159581 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:52.159624 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:52.160820 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:52.161288 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:52.161294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:52.161724 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:52.162167 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:52.162180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.162958 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.162961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:52.163814 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:52.163822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.164278 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.164281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:52.164415 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:52.165001 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:52.165018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:52.166689 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:52.168531 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:52.168536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:52.168547 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:52.168826 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:52.169116 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:52.169118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:52.169120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:52.169272 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:52.169776 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:52.169781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:52.170000 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 16:35:52.170162 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 16:35:52.170402 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260619 16:35:52.172255 4419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38801
I20260619 16:35:52.172489 4419 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:52.172984 4419 heartbeater.cc:511] Master 127.0.98.62:38801 requested a full tablet report, sending...
I20260619 16:35:52.174041 3960 ts_manager.cc:194] Registered new tserver with Master: 35afca84e2e54bfa8ccc807109ad418a (127.0.98.3:44751)
I20260619 16:35:52.174589 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36293
I20260619 16:35:52.179870 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:52.205446 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:52.195347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.0.98.1:43377)
0619 16:35:52.195688 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:52.195702 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:52.195958 (+ 256us) server_negotiation.cc:407] Connection header received
0619 16:35:52.196076 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:52.196081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:52.196284 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:52.196434 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:52.197155 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.198046 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.198864 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.199302 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.199451 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:52.201905 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:52.201913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:52.201925 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:52.201946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:52.203623 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:52.204063 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:52.204069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:52.204070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:52.204147 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:52.204425 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:52.204428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:52.204430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:52.204662 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:52.204892 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:52.205048 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:35:52.205202 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":222,"threads_started":1}
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:52.216351 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:52.208263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.0.98.62:38801)
0619 16:35:52.208388 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:52.208392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:52.208405 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:35:52.208483 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:52.208485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:52.208525 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:52.208600 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:52.209148 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.209705 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.210531 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:52.210793 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:52.210888 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:52.213151 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:52.213154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:52.213156 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:52.213179 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:52.214649 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:52.215063 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:52.215068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:52.215069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:52.215149 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:52.215423 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:52.215426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:52.215428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:52.215635 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:52.215776 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:52.215996 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 16:35:52.216125 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 16:35:52.216727 3960 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35500
I20260619 16:35:52.217362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4022
I20260619 16:35:52.223089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4156
I20260619 16:35:52.228348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4290
I20260619 16:35:52.233497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 3928
2026-06-19T16:35:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2118 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[4431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[4431](info): set up 2 sockets
Jun 19 16:35:52 dist-test-slave-0f28 krb5kdc[4431](info): commencing operation
krb5kdc: starting...
W20260619 16:35:53.305610 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:53 dist-test-slave-0f28 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:53.424906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:53.529912 4444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:53.530138 4444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:53.530176 4444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:53.533993 4444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:53.534035 4444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:53.534054 4444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:53.534070 4444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:53.534085 4444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:53.538933 4444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:53.539757 4444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:53.540623 4444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:35:53.548440 4444 server_base.cc:1034] running on GCE node
W20260619 16:35:53.548429 4449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:53.548440 4450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:53.548451 4452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:53.549122 4444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:53.550163 4444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:53.551330 4444 hybrid_clock.cc:648] HybridClock initialized: now 1781886953551285 us; error 57 us; skew 500 ppm
Jun 19 16:35:53 dist-test-slave-0f28 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:53.554208 4444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:53.554908 4444 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:53.554971 4444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:53.555168 4444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:53.557036 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "a725897879b548ef9b87bb11926653c3"
format_stamp: "Formatted at 2026-06-19 16:35:53 on dist-test-slave-0f28"
server_key: "c531c1d22c69946944d6e6b9e414b1f1"
server_key_iv: "5150a6f571ac5b5b57a564c9f62ea8a7"
server_key_version: "encryptionkey@0"
I20260619 16:35:53.557596 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "a725897879b548ef9b87bb11926653c3"
format_stamp: "Formatted at 2026-06-19 16:35:53 on dist-test-slave-0f28"
server_key: "c531c1d22c69946944d6e6b9e414b1f1"
server_key_iv: "5150a6f571ac5b5b57a564c9f62ea8a7"
server_key_version: "encryptionkey@0"
I20260619 16:35:53.560923 4444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:35:53.563166 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:53.564420 4444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:53.564528 4444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "a725897879b548ef9b87bb11926653c3"
format_stamp: "Formatted at 2026-06-19 16:35:53 on dist-test-slave-0f28"
server_key: "c531c1d22c69946944d6e6b9e414b1f1"
server_key_iv: "5150a6f571ac5b5b57a564c9f62ea8a7"
server_key_version: "encryptionkey@0"
I20260619 16:35:53.564663 4444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:53.574694 4444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:53.577512 4444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:53.577692 4444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:53.584255 4444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34867
I20260619 16:35:53.584486 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34867 every 8 connection(s)
I20260619 16:35:53.584858 4444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:53.587323 4511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:53.590065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4444
I20260619 16:35:53.590190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:53.590463 392 external_mini_cluster.cc:1428] Setting key ef1bebf80643be436efccc93ce3e9bdb
I20260619 16:35:53.592893 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: Bootstrap starting.
I20260619 16:35:53.595239 4511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:53.596094 4511 log.cc:824] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:53.598246 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: No bootstrap required, opened a new log
Jun 19 16:35:53 dist-test-slave-0f28 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:53.602177 4511 raft_consensus.cc:348] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a725897879b548ef9b87bb11926653c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260619 16:35:53.602427 4511 raft_consensus.cc:374] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:53.602509 4511 raft_consensus.cc:729] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a725897879b548ef9b87bb11926653c3, State: Initialized, Role: FOLLOWER
I20260619 16:35:53.602970 4511 consensus_queue.cc:260] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a725897879b548ef9b87bb11926653c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260619 16:35:53.603091 4511 raft_consensus.cc:388] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:53.603175 4511 raft_consensus.cc:482] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:53.603267 4511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:53.604080 4511 raft_consensus.cc:504] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a725897879b548ef9b87bb11926653c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260619 16:35:53.604414 4511 leader_election.cc:302] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a725897879b548ef9b87bb11926653c3; no voters:
I20260619 16:35:53.604686 4511 leader_election.cc:288] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:53.604774 4516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:53.604970 4516 raft_consensus.cc:686] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [term 1 LEADER]: Becoming Leader. State: Replica: a725897879b548ef9b87bb11926653c3, State: Running, Role: LEADER
I20260619 16:35:53.605208 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:53.591701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.0.98.62:34867)
0619 16:35:53.592067 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:53.592089 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:53.592122 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:35:53.592747 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:53.592765 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:53.593116 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:53.593437 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:53.594238 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:53.595204 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:53.596350 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:53.596802 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:53.596914 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:53.599552 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:53.599558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:53.599572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:53.599603 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:53.602095 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:53.602518 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:53.602523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:53.602526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:53.602675 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:53.602953 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:53.602958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:53.602959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:53.603479 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:53.603693 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:53.603959 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 16:35:53.604292 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":120,"threads_started":1}
I20260619 16:35:53.605458 4516 consensus_queue.cc:237] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a725897879b548ef9b87bb11926653c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260619 16:35:53.605826 4511 sys_catalog.cc:564] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:53.606972 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a725897879b548ef9b87bb11926653c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a725897879b548ef9b87bb11926653c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } }
I20260619 16:35:53.607089 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:53.607445 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a725897879b548ef9b87bb11926653c3. Latest consensus state: current_term: 1 leader_uuid: "a725897879b548ef9b87bb11926653c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a725897879b548ef9b87bb11926653c3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } }
I20260619 16:35:53.607585 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3 [sys.catalog]: This master's current role is: LEADER
W20260619 16:35:53.610189 4531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:35:53.610261 4531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:35:53.610320 4525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:53.611181 4525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:53.616036 4525 catalog_manager.cc:1347] Generated new cluster ID: 18c9b356d79448bf87c47498143f9468
I20260619 16:35:53.616094 4525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:53.624754 4525 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:53.625893 4525 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:53.630584 4525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: Generated new TSK 0
I20260619 16:35:53.631147 4525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:53.660727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:53.766569 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:53.766821 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:53.766893 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:53.770880 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:53.770963 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:53.771066 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:53.776093 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:53.777040 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:53.778013 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:53.786921 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:53.787045 4537 server_base.cc:1034] running on GCE node
W20260619 16:35:53.786935 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:53.786927 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:53.787591 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:53.788147 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:53.789389 4537 hybrid_clock.cc:648] HybridClock initialized: now 1781886953789380 us; error 28 us; skew 500 ppm
Jun 19 16:35:53 dist-test-slave-0f28 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:53.792073 4537 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:53.792761 4537 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:53.792853 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:53.793069 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:53.794993 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "5cd02dd72db3424da65e2472386fc8d8"
format_stamp: "Formatted at 2026-06-19 16:35:53 on dist-test-slave-0f28"
server_key: "536e4e88f6badd6f8b0b340bd316de76"
server_key_iv: "b516f4826915a0270caada58d4051a3b"
server_key_version: "encryptionkey@0"
I20260619 16:35:53.795535 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5cd02dd72db3424da65e2472386fc8d8"
format_stamp: "Formatted at 2026-06-19 16:35:53 on dist-test-slave-0f28"
server_key: "536e4e88f6badd6f8b0b340bd316de76"
server_key_iv: "b516f4826915a0270caada58d4051a3b"
server_key_version: "encryptionkey@0"
I20260619 16:35:53.799058 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 16:35:53.801304 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:53.802523 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:35:53.802680 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "5cd02dd72db3424da65e2472386fc8d8"
format_stamp: "Formatted at 2026-06-19 16:35:53 on dist-test-slave-0f28"
server_key: "536e4e88f6badd6f8b0b340bd316de76"
server_key_iv: "b516f4826915a0270caada58d4051a3b"
server_key_version: "encryptionkey@0"
I20260619 16:35:53.802791 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:53.841410 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:53.844683 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:53.844909 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:53.845543 4537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:53.846412 4537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:53.846501 4537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:53.846580 4537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:53.846621 4537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:53.856021 4537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39129
I20260619 16:35:53.856033 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39129 every 8 connection(s)
I20260619 16:35:53.856666 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:53.857149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4537
I20260619 16:35:53.857265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:53.857622 392 external_mini_cluster.cc:1428] Setting key 794464a2dc90f745a1211e21f93cf45c
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 16:35:53 dist-test-slave-0f28 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:53.873612 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:53.858774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59643 (local address 127.0.98.62:34867)
0619 16:35:53.858916 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:53.858923 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:53.859910 (+ 987us) server_negotiation.cc:407] Connection header received
0619 16:35:53.861393 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:53.861397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:53.861451 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:53.861550 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:53.862855 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:53.863441 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:53.864694 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:53.864943 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:53.865074 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:53.870045 (+ 4971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:53.870048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:53.870050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:53.870072 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:53.871568 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:53.872090 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:53.872096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:53.872098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:53.872177 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:53.872523 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:53.872527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:53.872529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:53.872708 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:53.872818 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:53.873293 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 16:35:53.873435 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:35:53.874382 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:53.859254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:59643)
0619 16:35:53.859765 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:53.859807 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:53.861160 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:53.861799 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:53.861806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:53.862188 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:53.862689 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:53.862704 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:53.863588 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:53.863592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:53.864553 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:53.864562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:53.865111 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:53.865116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:53.865295 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:53.865956 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:53.865972 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:53.869847 (+ 3875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:53.871706 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:53.871711 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:53.871725 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:53.871987 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:53.872277 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:53.872280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:53.872282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:53.872420 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:53.872817 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:53.872823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:53.873064 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 16:35:53.873252 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 16:35:53.873598 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":138,"threads_started":1}
I20260619 16:35:53.875705 4665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260619 16:35:53.875958 4665 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:53.876389 4665 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260619 16:35:53.877909 4475 ts_manager.cc:194] Registered new tserver with Master: 5cd02dd72db3424da65e2472386fc8d8 (127.0.98.1:39129)
I20260619 16:35:53.879086 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:53.884793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:53.992898 4670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:53.993125 4670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:53.993168 4670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:53.997097 4670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:53.997146 4670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:53.997203 4670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:54.002269 4670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:54.003144 4670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:54.004041 4670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:54.013358 4678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.013348 4675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.013409 4676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:54.013612 4670 server_base.cc:1034] running on GCE node
I20260619 16:35:54.014005 4670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:54.014596 4670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:54.015750 4670 hybrid_clock.cc:648] HybridClock initialized: now 1781886954015740 us; error 24 us; skew 500 ppm
Jun 19 16:35:54 dist-test-slave-0f28 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:54.018428 4670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:54.019142 4670 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:54.019232 4670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:54.019450 4670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:54.021468 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ca9d8bec15f48bca245cc9583d7bcac"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "e7483d12220ea2d7a14118bf45a817a3"
server_key_iv: "06bb6eac066c3718f93d8d051d6e9f1d"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.022006 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ca9d8bec15f48bca245cc9583d7bcac"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "e7483d12220ea2d7a14118bf45a817a3"
server_key_iv: "06bb6eac066c3718f93d8d051d6e9f1d"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.025368 4670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:54.027648 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.028859 4670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:54.029006 4670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "2ca9d8bec15f48bca245cc9583d7bcac"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "e7483d12220ea2d7a14118bf45a817a3"
server_key_iv: "06bb6eac066c3718f93d8d051d6e9f1d"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.029119 4670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:54.044135 4670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:54.047019 4670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:54.047210 4670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:54.047758 4670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:54.048514 4670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:54.048593 4670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.048664 4670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:54.048698 4670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.057686 4670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40697
I20260619 16:35:54.057710 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40697 every 8 connection(s)
I20260619 16:35:54.058215 4670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:54.060387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4670
I20260619 16:35:54.060577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:54.060945 392 external_mini_cluster.cc:1428] Setting key cd621738082488fd8b6b32956f823d89
Jun 19 16:35:54 dist-test-slave-0f28 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:54.072675 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.059883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59521 (local address 127.0.98.62:34867)
0619 16:35:54.060021 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.060024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.060897 (+ 873us) server_negotiation.cc:407] Connection header received
0619 16:35:54.062504 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.062510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.062561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.062638 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:54.064203 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.064725 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.065885 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.066295 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.066419 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.068832 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:54.068839 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:54.068842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:54.068876 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:54.070709 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:54.071227 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:54.071231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:54.071234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:54.071323 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:54.071645 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:54.071649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:54.071650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:54.071887 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:54.071985 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.072456 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.072559 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:35:54.073540 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.060214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:59521)
0619 16:35:54.060721 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.060779 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.062314 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.062824 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.062832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.063359 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:54.064033 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.064054 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.064837 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.064840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.065762 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.065770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.066447 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.066450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.066596 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.067169 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:54.067184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:54.068681 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:54.070835 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:54.070840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:54.070851 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:54.071121 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:54.071422 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:54.071424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:54.071426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:54.071553 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:54.072012 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:54.072018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:54.072239 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.072394 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.072638 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:35:54.074728 4798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260619 16:35:54.074988 4798 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:54.075428 4798 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260619 16:35:54.076508 4475 ts_manager.cc:194] Registered new tserver with Master: 2ca9d8bec15f48bca245cc9583d7bcac (127.0.98.2:40697)
I20260619 16:35:54.077096 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:54.090453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:54.199870 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:54.200274 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:54.200361 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:54.204494 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:54.204574 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:54.204671 4803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:54.210290 4803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:54.211254 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:54.212212 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:54.221357 4808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.221448 4811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.221357 4809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:54.221562 4803 server_base.cc:1034] running on GCE node
I20260619 16:35:54.221946 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:54.222592 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:54.223794 4803 hybrid_clock.cc:648] HybridClock initialized: now 1781886954223779 us; error 34 us; skew 500 ppm
Jun 19 16:35:54 dist-test-slave-0f28 krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:54.226622 4803 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:54.227276 4803 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:54.227336 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:54.227488 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:54.229478 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "e13880dfe39944d98bc3a8e2378c82d9"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "c752500370138299c2595a72d4cdffe4"
server_key_iv: "da625be7452defa05cbb831e8551bbed"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.229976 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e13880dfe39944d98bc3a8e2378c82d9"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "c752500370138299c2595a72d4cdffe4"
server_key_iv: "da625be7452defa05cbb831e8551bbed"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.233641 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:54.235898 4817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.237206 4803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:54.237298 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "e13880dfe39944d98bc3a8e2378c82d9"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "c752500370138299c2595a72d4cdffe4"
server_key_iv: "da625be7452defa05cbb831e8551bbed"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.237447 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:54.251791 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:54.254791 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:54.254976 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:54.255520 4803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:54.256356 4803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:54.256422 4803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.256489 4803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:54.256536 4803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.266026 4803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38123
I20260619 16:35:54.266043 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38123 every 8 connection(s)
I20260619 16:35:54.266566 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:35:54 dist-test-slave-0f28 krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:54.276517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4803
I20260619 16:35:54.276631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:54.276957 392 external_mini_cluster.cc:1428] Setting key ed787a295a39a8b3e8737058fee7d5ce
I20260619 16:35:54.279840 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.268196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38715 (local address 127.0.98.62:34867)
0619 16:35:54.268335 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.268339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.269246 (+ 907us) server_negotiation.cc:407] Connection header received
0619 16:35:54.270614 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.270617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.270689 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.270765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:54.271846 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.272364 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.273541 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.273771 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.273854 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.276185 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:54.276188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:54.276191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:54.276215 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:54.277890 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:54.278385 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:54.278390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:54.278392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:54.278471 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:54.278808 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:54.278813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:54.278815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:54.278988 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:54.279094 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.279555 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.279679 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:35:54.280553 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.268658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:38715)
0619 16:35:54.269104 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.269144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.270417 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.270920 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.270926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.271267 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:54.271717 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.271730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.272500 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.272503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.273426 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.273434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.273901 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.273903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.274033 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.274575 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:54.274591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:54.275996 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:54.277996 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:54.278002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:54.278015 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:54.278284 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:54.278569 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:54.278572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:54.278574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:54.278715 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:54.279091 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:54.279098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:54.279327 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.279504 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.279802 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260619 16:35:54.281620 4931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260619 16:35:54.281855 4931 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:54.282300 4931 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260619 16:35:54.283324 4475 ts_manager.cc:194] Registered new tserver with Master: e13880dfe39944d98bc3a8e2378c82d9 (127.0.98.3:38123)
I20260619 16:35:54.283897 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38715
I20260619 16:35:54.290836 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:35:54.300007 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.292728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.62:34867)
0619 16:35:54.292855 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.292859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.293025 (+ 166us) server_negotiation.cc:407] Connection header received
0619 16:35:54.293117 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.293119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.293154 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.293218 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:54.293963 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.294458 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.295258 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.295469 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.295549 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.296823 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:54.296826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:54.296828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:54.296847 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:54.298255 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:54.298683 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:54.298686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:54.298687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:54.298756 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:54.299025 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:54.299027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:54.299028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:54.299221 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:54.299320 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.299690 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.299841 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:35:54.302940 4475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:54.305970 4475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:54.321089 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.315225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.0.98.2:40697)
0619 16:35:54.315934 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.315946 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.316326 (+ 380us) server_negotiation.cc:407] Connection header received
0619 16:35:54.316711 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.316716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.316869 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.317063 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:54.317348 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.318253 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.319816 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.320448 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.320560 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.320696 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.320840 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.320936 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":156,"threads_started":1}
I20260619 16:35:54.321085 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.315095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40697 (local address 127.0.0.1:49190)
0619 16:35:54.316187 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.316239 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.316442 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.317055 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.317058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.317083 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:54.317209 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.317216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.318381 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.318387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.319673 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.319683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.320573 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.320576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.320649 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.320678 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.320793 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.320910 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":747,"spinlock_wait_cycles":5504,"thread_start_us":90,"threads_started":1}
I20260619 16:35:54.321795 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.315093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38123 (local address 127.0.0.1:33864)
0619 16:35:54.316186 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.316239 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.316442 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.317527 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.317530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.317549 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:54.317623 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.317628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.319311 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.319315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.320176 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.320183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.321377 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.321380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.321466 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.321502 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.321585 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.321668 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"spinlock_wait_cycles":17792,"thread_start_us":74,"threads_started":1}
I20260619 16:35:54.322245 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.315342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.0.98.3:38123)
0619 16:35:54.315933 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.315946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.316331 (+ 385us) server_negotiation.cc:407] Connection header received
0619 16:35:54.317165 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.317170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.317378 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.317536 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:54.317724 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.319174 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.320292 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.321060 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.321160 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.321284 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.322018 (+ 734us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.322121 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":81,"threads_started":1}
I20260619 16:35:54.323143 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.315334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.0.98.1:39129)
0619 16:35:54.315736 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.315746 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.316322 (+ 576us) server_negotiation.cc:407] Connection header received
0619 16:35:54.316626 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.316632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.316816 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.316936 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:54.317425 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.318569 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.320753 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.321567 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.321691 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.321822 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.322895 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.322992 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":150,"threads_started":1}
I20260619 16:35:54.324370 4732 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8a76db6352468a9f2b83e1a455ce1d (DEFAULT_TABLE table=test-table [id=7376a0d8fa834a3eb1a99293aa57af88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:54.325119 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.315093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39129 (local address 127.0.0.1:48638)
0619 16:35:54.316186 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.316239 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.316442 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.316999 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.317007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.317034 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:54.317215 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.317223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.318722 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.318725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.319686 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.319694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.322645 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.322649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.322746 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.322767 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.322847 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.322912 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":557,"spinlock_wait_cycles":25472,"thread_start_us":99,"threads_started":1}
I20260619 16:35:54.324811 4865 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8a76db6352468a9f2b83e1a455ce1d (DEFAULT_TABLE table=test-table [id=7376a0d8fa834a3eb1a99293aa57af88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:54.325696 4732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a76db6352468a9f2b83e1a455ce1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:54.325798 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a76db6352468a9f2b83e1a455ce1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:54.327415 4599 tablet_service.cc:1467] Processing CreateTablet for tablet 3d8a76db6352468a9f2b83e1a455ce1d (DEFAULT_TABLE table=test-table [id=7376a0d8fa834a3eb1a99293aa57af88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:54.328320 4599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d8a76db6352468a9f2b83e1a455ce1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:54.333309 4948 tablet_bootstrap.cc:492] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Bootstrap starting.
I20260619 16:35:54.333596 4949 tablet_bootstrap.cc:492] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Bootstrap starting.
I20260619 16:35:54.334158 4950 tablet_bootstrap.cc:492] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: Bootstrap starting.
I20260619 16:35:54.335839 4948 tablet_bootstrap.cc:654] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:54.335985 4950 tablet_bootstrap.cc:654] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:54.336412 4949 tablet_bootstrap.cc:654] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:54.337018 4950 log.cc:824] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:54.337124 4948 log.cc:824] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:54.337549 4949 log.cc:824] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:54.338908 4950 tablet_bootstrap.cc:492] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: No bootstrap required, opened a new log
I20260619 16:35:54.339115 4950 ts_tablet_manager.cc:1397] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:35:54.339252 4948 tablet_bootstrap.cc:492] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: No bootstrap required, opened a new log
I20260619 16:35:54.339252 4949 tablet_bootstrap.cc:492] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: No bootstrap required, opened a new log
I20260619 16:35:54.339428 4949 ts_tablet_manager.cc:1397] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:35:54.339428 4948 ts_tablet_manager.cc:1397] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:35:54.342067 4950 raft_consensus.cc:348] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.342296 4950 raft_consensus.cc:374] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:54.342348 4950 raft_consensus.cc:729] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd02dd72db3424da65e2472386fc8d8, State: Initialized, Role: FOLLOWER
I20260619 16:35:54.342803 4950 consensus_queue.cc:260] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.343173 4949 raft_consensus.cc:348] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.343173 4948 raft_consensus.cc:348] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.343423 4949 raft_consensus.cc:374] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:54.343431 4665 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
I20260619 16:35:54.343472 4949 raft_consensus.cc:729] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca9d8bec15f48bca245cc9583d7bcac, State: Initialized, Role: FOLLOWER
I20260619 16:35:54.343423 4948 raft_consensus.cc:374] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:54.343550 4948 raft_consensus.cc:729] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13880dfe39944d98bc3a8e2378c82d9, State: Initialized, Role: FOLLOWER
I20260619 16:35:54.343842 4950 ts_tablet_manager.cc:1428] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:35:54.343892 4948 consensus_queue.cc:260] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.343890 4949 consensus_queue.cc:260] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.344710 4931 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
I20260619 16:35:54.344792 4949 ts_tablet_manager.cc:1428] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:35:54.344792 4948 ts_tablet_manager.cc:1428] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:35:54.345109 4798 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
W20260619 16:35:54.357699 4666 tablet.cc:2369] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:54.398227 4956 raft_consensus.cc:482] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:54.398485 4956 raft_consensus.cc:504] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.399727 4956 leader_election.cc:288] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cd02dd72db3424da65e2472386fc8d8 (127.0.98.1:39129), 2ca9d8bec15f48bca245cc9583d7bcac (127.0.98.2:40697)
I20260619 16:35:54.403693 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.400048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40697 (local address 127.0.98.3:56737)
0619 16:35:54.400186 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.400203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.400301 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.400820 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.400823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.400841 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:54.400991 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.400997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.401818 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.401824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.402621 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.402630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.403271 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.403274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.403376 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.403404 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.403490 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.403562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:35:54.403693 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.400344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56737 (local address 127.0.98.2:40697)
0619 16:35:54.400577 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.400582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.400602 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:35:54.400664 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.400667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.400712 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.400808 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:54.401118 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.401709 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.402750 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.403154 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.403271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.403339 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.403514 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.403577 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 16:35:54.403944 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.400321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40927 (local address 127.0.98.1:39129)
0619 16:35:54.400626 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.400632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.400852 (+ 220us) server_negotiation.cc:407] Connection header received
0619 16:35:54.401007 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.401010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.401063 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.401166 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:35:54.401460 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.401935 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.402970 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.403361 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.403495 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.403554 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.403738 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.403824 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 16:35:54.404232 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.400048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39129 (local address 127.0.98.3:40927)
0619 16:35:54.400757 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.400768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.400894 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.401176 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.401179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.401194 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:35:54.401358 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.401367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.402052 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.402057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.402840 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:54.402848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.403503 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.403506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:54.403611 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.403631 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.403724 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.404095 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":326,"thread_start_us":125,"threads_started":1}
I20260619 16:35:54.404390 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a76db6352468a9f2b83e1a455ce1d" candidate_uuid: "e13880dfe39944d98bc3a8e2378c82d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" is_pre_election: true
I20260619 16:35:54.404722 4752 raft_consensus.cc:2454] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13880dfe39944d98bc3a8e2378c82d9 in term 0.
I20260619 16:35:54.404805 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a76db6352468a9f2b83e1a455ce1d" candidate_uuid: "e13880dfe39944d98bc3a8e2378c82d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd02dd72db3424da65e2472386fc8d8" is_pre_election: true
I20260619 16:35:54.405069 4619 raft_consensus.cc:2454] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e13880dfe39944d98bc3a8e2378c82d9 in term 0.
I20260619 16:35:54.405225 4821 leader_election.cc:302] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca9d8bec15f48bca245cc9583d7bcac, e13880dfe39944d98bc3a8e2378c82d9; no voters:
I20260619 16:35:54.405507 4956 raft_consensus.cc:2781] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:54.405604 4956 raft_consensus.cc:482] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:54.405665 4956 raft_consensus.cc:3037] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:54.406934 4956 raft_consensus.cc:504] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.407270 4956 leader_election.cc:288] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [CANDIDATE]: Term 1 election: Requested vote from peers 5cd02dd72db3424da65e2472386fc8d8 (127.0.98.1:39129), 2ca9d8bec15f48bca245cc9583d7bcac (127.0.98.2:40697)
I20260619 16:35:54.407630 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a76db6352468a9f2b83e1a455ce1d" candidate_uuid: "e13880dfe39944d98bc3a8e2378c82d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ca9d8bec15f48bca245cc9583d7bcac"
I20260619 16:35:54.407630 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d8a76db6352468a9f2b83e1a455ce1d" candidate_uuid: "e13880dfe39944d98bc3a8e2378c82d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cd02dd72db3424da65e2472386fc8d8"
I20260619 16:35:54.407749 4619 raft_consensus.cc:3037] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:54.407785 4752 raft_consensus.cc:3037] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:54.409037 4619 raft_consensus.cc:2454] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13880dfe39944d98bc3a8e2378c82d9 in term 1.
I20260619 16:35:54.409037 4752 raft_consensus.cc:2454] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e13880dfe39944d98bc3a8e2378c82d9 in term 1.
I20260619 16:35:54.409363 4819 leader_election.cc:302] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cd02dd72db3424da65e2472386fc8d8, e13880dfe39944d98bc3a8e2378c82d9; no voters:
I20260619 16:35:54.409601 4956 raft_consensus.cc:2781] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:54.409826 4956 raft_consensus.cc:686] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 1 LEADER]: Becoming Leader. State: Replica: e13880dfe39944d98bc3a8e2378c82d9, State: Running, Role: LEADER
I20260619 16:35:54.410169 4956 consensus_queue.cc:237] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } }
I20260619 16:35:54.413309 4474 catalog_manager.cc:5495] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e13880dfe39944d98bc3a8e2378c82d9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e13880dfe39944d98bc3a8e2378c82d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e13880dfe39944d98bc3a8e2378c82d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 } health_report { overall_health: UNKNOWN } } }
I20260619 16:35:54.446018 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.442658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.0.98.3:38123)
0619 16:35:54.442809 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.442812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.442825 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:35:54.442968 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.442970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.443024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.443116 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:35:54.443350 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.443849 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.444665 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:54.444871 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:54.444950 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:54.445214 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:35:54.445631 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:35:54.445725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.445779 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.445824 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:35:54.451785 4752 raft_consensus.cc:1261] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 1 FOLLOWER]: Refusing update from remote peer e13880dfe39944d98bc3a8e2378c82d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:54.451785 4619 raft_consensus.cc:1261] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 1 FOLLOWER]: Refusing update from remote peer e13880dfe39944d98bc3a8e2378c82d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:54.452436 4956 consensus_queue.cc:1035] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cd02dd72db3424da65e2472386fc8d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:54.452629 4958 consensus_queue.cc:1035] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca9d8bec15f48bca245cc9583d7bcac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:54.459026 4961 mvcc.cc:205] Tried to move back new op lower bound from 7298608965427621888 to 7298608965266784256. Current Snapshot: MvccSnapshot[applied={T|T < 7298608965427621888}]
I20260619 16:35:54.462611 4963 mvcc.cc:205] Tried to move back new op lower bound from 7298608965427621888 to 7298608965266784256. Current Snapshot: MvccSnapshot[applied={T|T < 7298608965427621888}]
I20260619 16:35:54.463289 4962 mvcc.cc:205] Tried to move back new op lower bound from 7298608965427621888 to 7298608965266784256. Current Snapshot: MvccSnapshot[applied={T|T < 7298608965427621888 or (T in {7298608965427621888})}]
I20260619 16:35:54.467630 4473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:54.467795 4473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:54.470342 4473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a725897879b548ef9b87bb11926653c3: Sending DeleteTablet for 3 replicas of tablet 3d8a76db6352468a9f2b83e1a455ce1d
I20260619 16:35:54.471081 4865 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8a76db6352468a9f2b83e1a455ce1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33864
I20260619 16:35:54.471211 4599 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8a76db6352468a9f2b83e1a455ce1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48638
I20260619 16:35:54.471724 4971 tablet_replica.cc:330] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8: stopping tablet replica
I20260619 16:35:54.471777 4970 tablet_replica.cc:330] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: stopping tablet replica
I20260619 16:35:54.472038 4971 raft_consensus.cc:2229] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:54.472095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4537
I20260619 16:35:54.472056 4970 raft_consensus.cc:2229] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:54.472339 4971 raft_consensus.cc:2258] T 3d8a76db6352468a9f2b83e1a455ce1d P 5cd02dd72db3424da65e2472386fc8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:54.472726 4732 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d8a76db6352468a9f2b83e1a455ce1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49190
I20260619 16:35:54.472875 4970 raft_consensus.cc:2258] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:54.473168 4972 tablet_replica.cc:330] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: stopping tablet replica
I20260619 16:35:54.473541 4972 raft_consensus.cc:2229] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:54.473870 4972 raft_consensus.cc:2258] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:54.474447 4970 ts_tablet_manager.cc:1905] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:54.475315 4972 ts_tablet_manager.cc:1905] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:54.477135 4970 ts_tablet_manager.cc:1918] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:35:54.477234 4970 log.cc:1196] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/3d8a76db6352468a9f2b83e1a455ce1d
I20260619 16:35:54.477599 4970 ts_tablet_manager.cc:1939] T 3d8a76db6352468a9f2b83e1a455ce1d P e13880dfe39944d98bc3a8e2378c82d9: Deleting consensus metadata
I20260619 16:35:54.478273 4972 ts_tablet_manager.cc:1918] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:35:54.478418 4972 log.cc:1196] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/3d8a76db6352468a9f2b83e1a455ce1d
I20260619 16:35:54.478889 4972 ts_tablet_manager.cc:1939] T 3d8a76db6352468a9f2b83e1a455ce1d P 2ca9d8bec15f48bca245cc9583d7bcac: Deleting consensus metadata
I20260619 16:35:54.479897 4462 catalog_manager.cc:4841] TS 2ca9d8bec15f48bca245cc9583d7bcac (127.0.98.2:40697): tablet 3d8a76db6352468a9f2b83e1a455ce1d (table test-table [id=7376a0d8fa834a3eb1a99293aa57af88]) successfully deleted
I20260619 16:35:54.481477 4459 catalog_manager.cc:4841] TS e13880dfe39944d98bc3a8e2378c82d9 (127.0.98.3:38123): tablet 3d8a76db6352468a9f2b83e1a455ce1d (table test-table [id=7376a0d8fa834a3eb1a99293aa57af88]) successfully deleted
W20260619 16:35:54.482343 4460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39129 (error 108)
I20260619 16:35:54.483120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4670
I20260619 16:35:54.483369 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.483094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39129 (local address 127.0.0.1:34856)
0619 16:35:54.483234 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.483296 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:35:54.483805 4460 catalog_manager.cc:4584] TS 5cd02dd72db3424da65e2472386fc8d8 (127.0.98.1:39129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3d8a76db6352468a9f2b83e1a455ce1d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39129: connect: Connection refused (error 111)
I20260619 16:35:54.489159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4803
I20260619 16:35:54.494467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4444
2026-06-19T16:35:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2260 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T16:35:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:54Z Disabled control of system clock
I20260619 16:35:54.523312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35549
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:35:54.628113 4977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:54.628350 4977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:54.628384 4977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:54.632207 4977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:54.632256 4977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:54.632272 4977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:54.632288 4977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:54.632303 4977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:54.636809 4977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35549
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:54.637802 4977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:54.639058 4977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:54.646890 4983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.647133 4985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:54.647255 4977 server_base.cc:1034] running on GCE node
W20260619 16:35:54.647282 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:54.647853 4977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:54.648829 4977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:54.649996 4977 hybrid_clock.cc:648] HybridClock initialized: now 1781886954649980 us; error 29 us; skew 500 ppm
I20260619 16:35:54.656129 4977 webserver.cc:460] Webserver started at http://127.0.98.62:39877/ using document root <none> and password file <none>
I20260619 16:35:54.656668 4977 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:54.656728 4977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:54.656930 4977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:54.658896 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "459bca1300d24983918c705405828a62"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "d9b77653402448a2b976e1481b2f4263"
server_key_iv: "9c6ff1bdadd535044743981ee288b7d0"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.659430 4977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "459bca1300d24983918c705405828a62"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "d9b77653402448a2b976e1481b2f4263"
server_key_iv: "9c6ff1bdadd535044743981ee288b7d0"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.662859 4977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 16:35:54.665119 4991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.666386 4977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:35:54.666524 4977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "459bca1300d24983918c705405828a62"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "d9b77653402448a2b976e1481b2f4263"
server_key_iv: "9c6ff1bdadd535044743981ee288b7d0"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.666631 4977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:54.671015 4977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:54.671582 4977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:54.671761 4977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:54.678084 4977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35549
I20260619 16:35:54.678149 5043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35549 every 8 connection(s)
I20260619 16:35:54.678951 4977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:54.681946 5044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:54.686941 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62: Bootstrap starting.
I20260619 16:35:54.688205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 4977
I20260619 16:35:54.688334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:54.688591 392 external_mini_cluster.cc:1428] Setting key f39d5c796a0e6288935ccb6231056849
I20260619 16:35:54.689375 5044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:54.690171 5044 log.cc:824] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:54.691998 5044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62: No bootstrap required, opened a new log
I20260619 16:35:54.693413 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.689627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.98.62:35549)
0619 16:35:54.690224 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.690238 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.690269 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:35:54.690892 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.690907 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.691174 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.691492 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:54.692208 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:54.692212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:54.692222 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:54.692232 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:54.692281 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:54.692296 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:54.692410 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.692507 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.692737 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260619 16:35:54.695788 5044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459bca1300d24983918c705405828a62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260619 16:35:54.696013 5044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:54.696089 5044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459bca1300d24983918c705405828a62, State: Initialized, Role: FOLLOWER
I20260619 16:35:54.696523 5044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459bca1300d24983918c705405828a62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260619 16:35:54.696668 5044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:54.696751 5044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:54.696843 5044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:54.697712 5044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459bca1300d24983918c705405828a62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260619 16:35:54.698017 5044 leader_election.cc:302] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 459bca1300d24983918c705405828a62; no voters:
I20260619 16:35:54.698276 5044 leader_election.cc:288] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:54.698380 5049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:54.698655 5049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [term 1 LEADER]: Becoming Leader. State: Replica: 459bca1300d24983918c705405828a62, State: Running, Role: LEADER
I20260619 16:35:54.698922 5049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459bca1300d24983918c705405828a62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } }
I20260619 16:35:54.699375 5044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:54.700578 5050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "459bca1300d24983918c705405828a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459bca1300d24983918c705405828a62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } } }
I20260619 16:35:54.700623 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 459bca1300d24983918c705405828a62. Latest consensus state: current_term: 1 leader_uuid: "459bca1300d24983918c705405828a62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459bca1300d24983918c705405828a62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35549 } } }
I20260619 16:35:54.700695 5050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:54.700711 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:54.701289 5058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:54.703806 5058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:54.708696 5058 catalog_manager.cc:1347] Generated new cluster ID: 0ce451314e744c09bee8d4038a5485a1
I20260619 16:35:54.708756 5058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:54.724584 5058 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:54.724951 5058 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:54.732024 5058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62: Generated new TSK 0
I20260619 16:35:54.732710 5058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:35:54.743920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35549
--builtin_ntp_servers=127.0.98.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 16:35:54.852905 5068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:54.853178 5068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:54.853248 5068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:54.857304 5068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:54.857390 5068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:54.857474 5068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:54.862320 5068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:54.863435 5068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:54.864861 5068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:54.875608 5074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.875641 5076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:54.875640 5073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:54.875958 5068 server_base.cc:1034] running on GCE node
I20260619 16:35:54.876330 5068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:54.876945 5068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:54.878114 5068 hybrid_clock.cc:648] HybridClock initialized: now 1781886954878078 us; error 45 us; skew 500 ppm
I20260619 16:35:54.879915 5068 webserver.cc:460] Webserver started at http://127.0.98.1:34051/ using document root <none> and password file <none>
I20260619 16:35:54.880453 5068 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:54.880518 5068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:54.880707 5068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:54.882669 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee67acc4476641b19d927fdc058ccc59"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "31449a529ab55ac43ff2df3e1b1197c8"
server_key_iv: "d8a427119a107365ac4a40aa297bcb34"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.883183 5068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee67acc4476641b19d927fdc058ccc59"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "31449a529ab55ac43ff2df3e1b1197c8"
server_key_iv: "d8a427119a107365ac4a40aa297bcb34"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.886945 5068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:35:54.889261 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.890560 5068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:54.890722 5068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "ee67acc4476641b19d927fdc058ccc59"
format_stamp: "Formatted at 2026-06-19 16:35:54 on dist-test-slave-0f28"
server_key: "31449a529ab55ac43ff2df3e1b1197c8"
server_key_iv: "d8a427119a107365ac4a40aa297bcb34"
server_key_version: "encryptionkey@0"
I20260619 16:35:54.890831 5068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:54.895201 5068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:54.895756 5068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:54.895932 5068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:54.896430 5068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:54.897287 5068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:54.897395 5068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.897475 5068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:54.897531 5068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:54.906479 5068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43711
I20260619 16:35:54.906502 5195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43711 every 8 connection(s)
I20260619 16:35:54.907322 5068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:54.909864 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5068
I20260619 16:35:54.909979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:54.910257 392 external_mini_cluster.cc:1428] Setting key 1b6eb078b09f70ee15d8f514313bbde2
I20260619 16:35:54.912153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35549
--builtin_ntp_servers=127.0.98.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:35:54.913843 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.909001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41467 (local address 127.0.98.62:35549)
0619 16:35:54.909156 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:54.909161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:54.910606 (+ 1445us) server_negotiation.cc:407] Connection header received
0619 16:35:54.912004 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:54.912009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:54.912053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:54.912173 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:54.913360 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:54.913364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:54.913366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:54.913379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:54.913399 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:54.913415 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:54.913529 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:54.913673 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 16:35:54.913716 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:35:54.914644 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:54.909471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35549 (local address 127.0.98.1:41467)
0619 16:35:54.910407 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:54.910446 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:54.911806 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:54.912366 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:54.912373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:54.912456 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:54.913088 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:54.913089 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:54.913127 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:54.913190 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:54.913516 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:54.913524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:54.913530 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:35:54.913702 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 16:35:54.913926 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":186,"threads_started":1}
I20260619 16:35:54.915825 5196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35549
I20260619 16:35:54.916067 5196 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:54.916548 5196 heartbeater.cc:511] Master 127.0.98.62:35549 requested a full tablet report, sending...
I20260619 16:35:54.918138 5008 ts_manager.cc:194] Registered new tserver with Master: ee67acc4476641b19d927fdc058ccc59 (127.0.98.1:43711)
W20260619 16:35:55.022353 5199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:55.022596 5199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:55.022634 5199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:55.026684 5199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:55.026741 5199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:55.026794 5199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:55.031462 5199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:55.032496 5199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:55.033953 5199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:55.043161 5204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:55.043203 5205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:55.043203 5207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:55.043242 5199 server_base.cc:1034] running on GCE node
I20260619 16:35:55.043733 5199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:55.044381 5199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:55.045591 5199 hybrid_clock.cc:648] HybridClock initialized: now 1781886955045578 us; error 33 us; skew 500 ppm
I20260619 16:35:55.047498 5199 webserver.cc:460] Webserver started at http://127.0.98.2:37141/ using document root <none> and password file <none>
I20260619 16:35:55.048063 5199 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:55.048156 5199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:55.048381 5199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:55.050369 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "15c4de3e297a43de85d11d5c30d34d89"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "32be2d53d03a1c1006b4688554bf4ee6"
server_key_iv: "b07ad07f4d38974d43d08d5cf2dab94c"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.050897 5199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "15c4de3e297a43de85d11d5c30d34d89"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "32be2d53d03a1c1006b4688554bf4ee6"
server_key_iv: "b07ad07f4d38974d43d08d5cf2dab94c"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.054373 5199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:55.056744 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.057983 5199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:55.058120 5199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "15c4de3e297a43de85d11d5c30d34d89"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "32be2d53d03a1c1006b4688554bf4ee6"
server_key_iv: "b07ad07f4d38974d43d08d5cf2dab94c"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.058228 5199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:55.062546 5199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:55.063114 5199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:55.063288 5199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:55.063804 5199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:55.064647 5199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:55.064718 5199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.064783 5199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:55.064829 5199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.074043 5199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34261
I20260619 16:35:55.074065 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34261 every 8 connection(s)
I20260619 16:35:55.074870 5199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:55.078177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5199
I20260619 16:35:55.078297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:55.078639 392 external_mini_cluster.cc:1428] Setting key 18940779fa10363a2c9e42af7e9564cc
I20260619 16:35:55.080827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35549
--builtin_ntp_servers=127.0.98.20:39429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 16:35:55.081750 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.076702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51257 (local address 127.0.98.62:35549)
0619 16:35:55.076854 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.076858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.078108 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 16:35:55.079748 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.079752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.079785 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.079875 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.081180 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.081187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.081193 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.081210 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.081237 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.081250 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.081377 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.081515 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.081562 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:35:55.082463 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.077163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35549 (local address 127.0.98.2:51257)
0619 16:35:55.077860 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.077963 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.079546 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.080089 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.080096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.080217 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.080851 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.080853 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.080918 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.081017 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.081372 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.081385 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.081390 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.081518 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.081702 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":164,"threads_started":1}
I20260619 16:35:55.083618 5327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35549
I20260619 16:35:55.083844 5327 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:55.084229 5327 heartbeater.cc:511] Master 127.0.98.62:35549 requested a full tablet report, sending...
I20260619 16:35:55.085273 5008 ts_manager.cc:194] Registered new tserver with Master: 15c4de3e297a43de85d11d5c30d34d89 (127.0.98.2:34261)
W20260619 16:35:55.188436 5330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:55.188671 5330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:55.188709 5330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:55.192652 5330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:55.192703 5330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:55.192756 5330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:55.197954 5330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:55.198984 5330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:55.200521 5330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:55.209990 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:55.210045 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:55.210253 5330 server_base.cc:1034] running on GCE node
W20260619 16:35:55.210284 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:55.210650 5330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:55.211220 5330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:55.212401 5330 hybrid_clock.cc:648] HybridClock initialized: now 1781886955212345 us; error 74 us; skew 500 ppm
I20260619 16:35:55.214227 5330 webserver.cc:460] Webserver started at http://127.0.98.3:34951/ using document root <none> and password file <none>
I20260619 16:35:55.214775 5330 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:55.214864 5330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:55.215091 5330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:55.217096 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "d98fce82162c4764bfa4e920c97cdd51"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "72219cadb8c8e3cc65a82ad11d423512"
server_key_iv: "0280f51a4a189965afb76d7bae456b4c"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.217680 5330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d98fce82162c4764bfa4e920c97cdd51"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "72219cadb8c8e3cc65a82ad11d423512"
server_key_iv: "0280f51a4a189965afb76d7bae456b4c"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.221524 5330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:35:55.223745 5344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.224830 5330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:55.224956 5330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "d98fce82162c4764bfa4e920c97cdd51"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "72219cadb8c8e3cc65a82ad11d423512"
server_key_iv: "0280f51a4a189965afb76d7bae456b4c"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.225066 5330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:55.229461 5330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:55.230012 5330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:55.230183 5330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:55.230682 5330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:55.231490 5330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:55.231555 5330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.231614 5330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:55.231664 5330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.240299 5330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37613
I20260619 16:35:55.240325 5457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37613 every 8 connection(s)
I20260619 16:35:55.241087 5330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:55.246809 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.242776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34175 (local address 127.0.98.62:35549)
0619 16:35:55.242908 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.242912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.243874 (+ 962us) server_negotiation.cc:407] Connection header received
0619 16:35:55.245134 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.245138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.245181 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.245259 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.246335 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.246338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.246341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.246352 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.246372 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.246381 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.246493 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.246618 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.246660 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:35:55.247037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5330
I20260619 16:35:55.247175 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:55.247438 392 external_mini_cluster.cc:1428] Setting key 580bb68792e2c9e64f8200fb37681f38
I20260619 16:35:55.247690 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.243149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35549 (local address 127.0.98.3:34175)
0619 16:35:55.243741 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.243777 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.244921 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.245438 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.245445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.245527 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.246092 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.246094 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.246130 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.246191 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.246472 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.246480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.246485 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.246657 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.246872 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260619 16:35:55.248713 5458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35549
I20260619 16:35:55.248927 5458 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:55.249379 5458 heartbeater.cc:511] Master 127.0.98.62:35549 requested a full tablet report, sending...
I20260619 16:35:55.250290 5008 ts_manager.cc:194] Registered new tserver with Master: d98fce82162c4764bfa4e920c97cdd51 (127.0.98.3:37613)
I20260619 16:35:55.261088 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:35:55.264753 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.263018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.0.98.62:35549)
0619 16:35:55.263160 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.263165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.263325 (+ 160us) server_negotiation.cc:407] Connection header received
0619 16:35:55.263464 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.263468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.263505 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.263654 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.264374 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.264378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.264380 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.264392 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.264410 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.264418 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.264515 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.264595 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.264633 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:35:55.267824 5008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:35:55.271406 5008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:35:55.282672 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.279661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37613 (local address 127.0.0.1:33298)
0619 16:35:55.280448 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.280460 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.280555 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.280983 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.280989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.281063 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.281607 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.281609 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.281628 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.281688 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.282371 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.282375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.282378 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.282438 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.282512 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":72,"threads_started":1}
I20260619 16:35:55.282706 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.279395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.0.98.1:43711)
0619 16:35:55.279815 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.279827 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.280227 (+ 400us) server_negotiation.cc:407] Connection header received
0619 16:35:55.280374 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.280380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.280579 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.280691 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.282142 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.282148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.282160 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.282171 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.282218 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.282240 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.282348 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.282464 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.282535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":111,"threads_started":1}
I20260619 16:35:55.282958 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.280052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.98.3:37613)
0619 16:35:55.280416 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.280428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.280544 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:35:55.280641 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.280649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.280822 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.281347 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.282180 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.282184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.282192 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.282202 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.282238 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.282260 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.282597 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.282643 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.282705 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260619 16:35:55.282994 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.279399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34261 (local address 127.0.0.1:46570)
0619 16:35:55.280133 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.280144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.280241 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.281355 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.281358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.281373 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.281827 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.281830 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.281841 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.281874 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.282371 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.282375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.282378 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.282437 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.282517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":78,"threads_started":1}
I20260619 16:35:55.283317 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.280401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.0.98.2:34261)
0619 16:35:55.280939 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.280951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.280982 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:35:55.281029 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.281033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.281188 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.281560 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.282176 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.282182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.282192 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.282202 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.282238 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.282260 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.282597 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.282643 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.282705 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":95,"threads_started":1}
I20260619 16:35:55.283349 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.279258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43711 (local address 127.0.0.1:52158)
0619 16:35:55.280070 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.280117 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.280229 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.281432 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.281437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.281456 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.281996 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.281997 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.282006 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.282031 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.282334 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.282343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.282348 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.282452 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.282507 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":79,"threads_started":1}
I20260619 16:35:55.285810 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 97517e73f8ff4940ae953e7409fcae6d (DEFAULT_TABLE table=test-table [id=04598ba15e164f1782a3063c58b1f2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:55.285810 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 97517e73f8ff4940ae953e7409fcae6d (DEFAULT_TABLE table=test-table [id=04598ba15e164f1782a3063c58b1f2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:55.286882 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97517e73f8ff4940ae953e7409fcae6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:55.286882 5392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97517e73f8ff4940ae953e7409fcae6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:55.287101 5116 tablet_service.cc:1467] Processing CreateTablet for tablet 97517e73f8ff4940ae953e7409fcae6d (DEFAULT_TABLE table=test-table [id=04598ba15e164f1782a3063c58b1f2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:35:55.287901 5116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97517e73f8ff4940ae953e7409fcae6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:55.293830 5477 tablet_bootstrap.cc:492] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: Bootstrap starting.
I20260619 16:35:55.294950 5476 tablet_bootstrap.cc:492] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Bootstrap starting.
I20260619 16:35:55.294950 5475 tablet_bootstrap.cc:492] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: Bootstrap starting.
I20260619 16:35:55.296042 5477 tablet_bootstrap.cc:654] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:55.296733 5477 log.cc:824] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:55.297043 5476 tablet_bootstrap.cc:654] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:55.297043 5475 tablet_bootstrap.cc:654] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:55.297888 5476 log.cc:824] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:55.297888 5475 log.cc:824] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:55.298732 5477 tablet_bootstrap.cc:492] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: No bootstrap required, opened a new log
I20260619 16:35:55.298923 5477 ts_tablet_manager.cc:1397] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:35:55.299599 5475 tablet_bootstrap.cc:492] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: No bootstrap required, opened a new log
I20260619 16:35:55.299777 5475 ts_tablet_manager.cc:1397] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:35:55.302582 5477 raft_consensus.cc:348] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.302806 5477 raft_consensus.cc:374] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:55.302856 5477 raft_consensus.cc:729] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c4de3e297a43de85d11d5c30d34d89, State: Initialized, Role: FOLLOWER
I20260619 16:35:55.303022 5476 tablet_bootstrap.cc:492] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: No bootstrap required, opened a new log
I20260619 16:35:55.303189 5476 ts_tablet_manager.cc:1397] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 16:35:55.303332 5477 consensus_queue.cc:260] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.303627 5475 raft_consensus.cc:348] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.303856 5475 raft_consensus.cc:374] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:55.303903 5475 raft_consensus.cc:729] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee67acc4476641b19d927fdc058ccc59, State: Initialized, Role: FOLLOWER
I20260619 16:35:55.304320 5477 ts_tablet_manager.cc:1428] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:35:55.304297 5475 consensus_queue.cc:260] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.304654 5327 heartbeater.cc:503] Master 127.0.98.62:35549 was elected leader, sending a full tablet report...
I20260619 16:35:55.305174 5475 ts_tablet_manager.cc:1428] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:35:55.305495 5196 heartbeater.cc:503] Master 127.0.98.62:35549 was elected leader, sending a full tablet report...
I20260619 16:35:55.307900 5476 raft_consensus.cc:348] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.308142 5476 raft_consensus.cc:374] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:55.308202 5476 raft_consensus.cc:729] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98fce82162c4764bfa4e920c97cdd51, State: Initialized, Role: FOLLOWER
I20260619 16:35:55.308580 5476 consensus_queue.cc:260] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.309175 5458 heartbeater.cc:503] Master 127.0.98.62:35549 was elected leader, sending a full tablet report...
I20260619 16:35:55.309367 5476 ts_tablet_manager.cc:1428] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260619 16:35:55.325716 5328 tablet.cc:2369] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:35:55.408396 5197 tablet.cc:2369] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:55.457794 5481 raft_consensus.cc:482] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:35:55.457990 5481 raft_consensus.cc:504] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.459091 5481 leader_election.cc:288] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee67acc4476641b19d927fdc058ccc59 (127.0.98.1:43711), d98fce82162c4764bfa4e920c97cdd51 (127.0.98.3:37613)
I20260619 16:35:55.461122 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.459480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53795 (local address 127.0.98.1:43711)
0619 16:35:55.459619 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.459624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.459644 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:35:55.459774 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.459777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.459825 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.459907 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.460698 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.460703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.460708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.460725 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.460750 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.460760 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.460885 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.460924 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.460966 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 16:35:55.461133 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.459374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43711 (local address 127.0.98.2:53795)
0619 16:35:55.459512 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.459529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.459642 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.459955 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.459959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.459975 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.460524 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.460526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.460539 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.460565 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.460846 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.460852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.460859 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.460926 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.460958 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 16:35:55.461292 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.459473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40703 (local address 127.0.98.3:37613)
0619 16:35:55.459625 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.459630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.459996 (+ 366us) server_negotiation.cc:407] Connection header received
0619 16:35:55.460086 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.460088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.460119 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.460219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.460916 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.460920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.460922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.460935 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.460957 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.460968 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.461052 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.461128 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.461170 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:35:55.461359 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.459374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37613 (local address 127.0.98.2:40703)
0619 16:35:55.459878 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.459911 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.460000 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.460254 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.460256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.460268 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:55.460754 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:35:55.460756 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:55.460763 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:35:55.460780 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:55.461049 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:55.461052 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:55.461055 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 16:35:55.461129 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:35:55.461163 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":144,"thread_start_us":121,"threads_started":1}
I20260619 16:35:55.461938 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517e73f8ff4940ae953e7409fcae6d" candidate_uuid: "15c4de3e297a43de85d11d5c30d34d89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee67acc4476641b19d927fdc058ccc59" is_pre_election: true
I20260619 16:35:55.461973 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517e73f8ff4940ae953e7409fcae6d" candidate_uuid: "15c4de3e297a43de85d11d5c30d34d89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98fce82162c4764bfa4e920c97cdd51" is_pre_election: true
I20260619 16:35:55.462227 5150 raft_consensus.cc:2454] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15c4de3e297a43de85d11d5c30d34d89 in term 0.
I20260619 16:35:55.462251 5412 raft_consensus.cc:2454] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15c4de3e297a43de85d11d5c30d34d89 in term 0.
I20260619 16:35:55.462678 5216 leader_election.cc:302] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c4de3e297a43de85d11d5c30d34d89, ee67acc4476641b19d927fdc058ccc59; no voters:
I20260619 16:35:55.462954 5481 raft_consensus.cc:2781] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:35:55.463042 5481 raft_consensus.cc:482] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:35:55.463073 5481 raft_consensus.cc:3037] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:55.464068 5481 raft_consensus.cc:504] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.464413 5481 leader_election.cc:288] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [CANDIDATE]: Term 1 election: Requested vote from peers ee67acc4476641b19d927fdc058ccc59 (127.0.98.1:43711), d98fce82162c4764bfa4e920c97cdd51 (127.0.98.3:37613)
I20260619 16:35:55.464704 5150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517e73f8ff4940ae953e7409fcae6d" candidate_uuid: "15c4de3e297a43de85d11d5c30d34d89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee67acc4476641b19d927fdc058ccc59"
I20260619 16:35:55.464726 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517e73f8ff4940ae953e7409fcae6d" candidate_uuid: "15c4de3e297a43de85d11d5c30d34d89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98fce82162c4764bfa4e920c97cdd51"
I20260619 16:35:55.464830 5150 raft_consensus.cc:3037] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:55.464843 5412 raft_consensus.cc:3037] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:55.465937 5412 raft_consensus.cc:2454] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15c4de3e297a43de85d11d5c30d34d89 in term 1.
I20260619 16:35:55.465942 5150 raft_consensus.cc:2454] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15c4de3e297a43de85d11d5c30d34d89 in term 1.
I20260619 16:35:55.466216 5215 leader_election.cc:302] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c4de3e297a43de85d11d5c30d34d89, d98fce82162c4764bfa4e920c97cdd51; no voters:
I20260619 16:35:55.466452 5481 raft_consensus.cc:2781] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:55.466681 5481 raft_consensus.cc:686] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 1 LEADER]: Becoming Leader. State: Replica: 15c4de3e297a43de85d11d5c30d34d89, State: Running, Role: LEADER
I20260619 16:35:55.467212 5481 consensus_queue.cc:237] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } }
I20260619 16:35:55.470201 5007 catalog_manager.cc:5495] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15c4de3e297a43de85d11d5c30d34d89 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "15c4de3e297a43de85d11d5c30d34d89" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15c4de3e297a43de85d11d5c30d34d89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 } health_report { overall_health: UNKNOWN } } }
W20260619 16:35:55.492065 5459 tablet.cc:2369] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:35:55.518553 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.516970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.0.98.2:34261)
0619 16:35:55.517125 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.517130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.517147 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:35:55.517266 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.517269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.517347 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.517444 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.518114 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.518118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.518120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.518132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.518158 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.518166 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.518256 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.518365 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.518404 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:35:55.524070 5412 raft_consensus.cc:1261] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 1 FOLLOWER]: Refusing update from remote peer 15c4de3e297a43de85d11d5c30d34d89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:55.524072 5150 raft_consensus.cc:1261] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 1 FOLLOWER]: Refusing update from remote peer 15c4de3e297a43de85d11d5c30d34d89: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:35:55.524788 5485 consensus_queue.cc:1035] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98fce82162c4764bfa4e920c97cdd51" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:55.525017 5481 consensus_queue.cc:1035] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee67acc4476641b19d927fdc058ccc59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:35:55.531268 5488 mvcc.cc:205] Tried to move back new op lower bound from 7298608969819967488 to 7298608969595723776. Current Snapshot: MvccSnapshot[applied={T|T < 7298608969819967488}]
I20260619 16:35:55.533135 5490 mvcc.cc:205] Tried to move back new op lower bound from 7298608969819967488 to 7298608969595723776. Current Snapshot: MvccSnapshot[applied={T|T < 7298608969819967488}]
I20260619 16:35:55.534049 5489 mvcc.cc:205] Tried to move back new op lower bound from 7298608969819967488 to 7298608969595723776. Current Snapshot: MvccSnapshot[applied={T|T < 7298608969819967488}]
I20260619 16:35:55.539464 5008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:55.539639 5008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:35:55.541855 5008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 459bca1300d24983918c705405828a62: Sending DeleteTablet for 3 replicas of tablet 97517e73f8ff4940ae953e7409fcae6d
I20260619 16:35:55.542543 5116 tablet_service.cc:1514] Processing DeleteTablet for tablet 97517e73f8ff4940ae953e7409fcae6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:55 UTC) from {username='slave'} at 127.0.0.1:52158
I20260619 16:35:55.542551 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet 97517e73f8ff4940ae953e7409fcae6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:55 UTC) from {username='slave'} at 127.0.0.1:46570
I20260619 16:35:55.543010 5498 tablet_replica.cc:330] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59: stopping tablet replica
I20260619 16:35:55.543022 5392 tablet_service.cc:1514] Processing DeleteTablet for tablet 97517e73f8ff4940ae953e7409fcae6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:35:55 UTC) from {username='slave'} at 127.0.0.1:33298
I20260619 16:35:55.543318 5498 raft_consensus.cc:2229] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:55.543381 5500 tablet_replica.cc:330] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: stopping tablet replica
I20260619 16:35:55.543599 5498 raft_consensus.cc:2258] T 97517e73f8ff4940ae953e7409fcae6d P ee67acc4476641b19d927fdc058ccc59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:55.543622 5500 raft_consensus.cc:2229] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:35:55.543682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5068
I20260619 16:35:55.543852 5500 raft_consensus.cc:2258] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:55.544853 5499 tablet_replica.cc:330] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89: stopping tablet replica
I20260619 16:35:55.545071 5499 raft_consensus.cc:2229] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:35:55.545384 5500 ts_tablet_manager.cc:1905] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:35:55.545421 5499 raft_consensus.cc:2258] T 97517e73f8ff4940ae953e7409fcae6d P 15c4de3e297a43de85d11d5c30d34d89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:35:55.548720 5500 ts_tablet_manager.cc:1918] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:35:55.548818 5500 log.cc:1196] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/97517e73f8ff4940ae953e7409fcae6d
I20260619 16:35:55.549157 5500 ts_tablet_manager.cc:1939] T 97517e73f8ff4940ae953e7409fcae6d P d98fce82162c4764bfa4e920c97cdd51: Deleting consensus metadata
I20260619 16:35:55.550349 4993 catalog_manager.cc:4841] TS d98fce82162c4764bfa4e920c97cdd51 (127.0.98.3:37613): tablet 97517e73f8ff4940ae953e7409fcae6d (table test-table [id=04598ba15e164f1782a3063c58b1f2b6]) successfully deleted
W20260619 16:35:55.551607 4994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43711 (error 108)
I20260619 16:35:55.551720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5199
I20260619 16:35:55.553267 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.552919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43711 (local address 127.0.0.1:52170)
0619 16:35:55.553082 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.553170 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 16:35:55.553524 4994 catalog_manager.cc:4584] TS ee67acc4476641b19d927fdc058ccc59 (127.0.98.1:43711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97517e73f8ff4940ae953e7409fcae6d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43711: connect: Connection refused (error 111)
I20260619 16:35:55.557580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5330
I20260619 16:35:55.558619 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.557786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34261 (local address 127.0.0.1:46594)
0619 16:35:55.558450 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.558512 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575}
I20260619 16:35:55.562865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 4977
2026-06-19T16:35:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1068 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 16:35:55.584229 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:35:55.585794 392 webserver.cc:460] Webserver started at https://127.0.0.1:42667/ using document root <none> and password file <none>
I20260619 16:35:55.585922 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:35:55.586136 392 webserver.cc:460] Webserver started at http://127.0.0.1:39795/ using document root <none> and password file <none>
2026-06-19T16:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:55Z Disabled control of system clock
I20260619 16:35:55.594825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45799
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45799
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42667/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/testchainca.cert with env {}
W20260619 16:35:55.698666 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:55.698961 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:55.699004 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:55.702819 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:55.702862 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:55.702876 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:55.702891 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:55.702906 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:35:55.702921 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:35:55.702934 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42667/jwks.json
I20260619 16:35:55.707856 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42667/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:55.708724 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:55.709653 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:55.717435 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:55.717470 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:55.717449 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:55.717662 5507 server_base.cc:1034] running on GCE node
I20260619 16:35:55.718267 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:55.719308 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:55.720517 5507 hybrid_clock.cc:648] HybridClock initialized: now 1781886955720501 us; error 32 us; skew 500 ppm
I20260619 16:35:55.722364 5507 webserver.cc:460] Webserver started at http://127.0.98.62:41341/ using document root <none> and password file <none>
I20260619 16:35:55.722905 5507 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:55.723004 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:55.723217 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:55.725111 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "f407d5a40c284d3eadde84d1d7f8f646"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "147e9f2d7293418924c38f7c26007421"
server_key_iv: "7677df0d93f313c9b06f6efb0999816f"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.725697 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "f407d5a40c284d3eadde84d1d7f8f646"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "147e9f2d7293418924c38f7c26007421"
server_key_iv: "7677df0d93f313c9b06f6efb0999816f"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.729059 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:55.731377 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:55.732517 5507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:55.732661 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "f407d5a40c284d3eadde84d1d7f8f646"
format_stamp: "Formatted at 2026-06-19 16:35:55 on dist-test-slave-0f28"
server_key: "147e9f2d7293418924c38f7c26007421"
server_key_iv: "7677df0d93f313c9b06f6efb0999816f"
server_key_version: "encryptionkey@0"
I20260619 16:35:55.732772 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:55.766137 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:55.766816 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:55.767010 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:55.773757 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45799
I20260619 16:35:55.773772 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45799 every 8 connection(s)
I20260619 16:35:55.774650 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:55.777767 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:55.780107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5507
I20260619 16:35:55.780234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:55.780519 392 external_mini_cluster.cc:1428] Setting key 3e54b50758b96ba30ee9a5560c2a5e0b
I20260619 16:35:55.783527 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646: Bootstrap starting.
I20260619 16:35:55.785936 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:55.786657 5574 log.cc:824] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:55.788715 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646: No bootstrap required, opened a new log
I20260619 16:35:55.789710 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.781774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.0.98.62:45799)
0619 16:35:55.782335 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.782351 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.782383 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:35:55.782985 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.783000 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.783361 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.783657 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:55.784013 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:55.784905 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:55.786147 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:55.786696 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:55.786855 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:55.787752 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:55.787758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:55.787773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:55.788236 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:55.788301 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:55.788326 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:55.788428 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:55.788569 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 16:35:55.788821 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":133,"threads_started":1}
I20260619 16:35:55.792284 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:35:55.792392 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f407d5a40c284d3eadde84d1d7f8f646" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45799 } }
I20260619 16:35:55.792656 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:55.792707 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f407d5a40c284d3eadde84d1d7f8f646, State: Initialized, Role: FOLLOWER
I20260619 16:35:55.793136 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f407d5a40c284d3eadde84d1d7f8f646" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45799 } }
I20260619 16:35:55.793291 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:55.793391 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:55.793498 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:55.794420 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f407d5a40c284d3eadde84d1d7f8f646" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45799 } }
I20260619 16:35:55.794749 5574 leader_election.cc:302] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f407d5a40c284d3eadde84d1d7f8f646; no voters:
I20260619 16:35:55.795019 5574 leader_election.cc:288] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:55.795141 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:55.795387 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [term 1 LEADER]: Becoming Leader. State: Replica: f407d5a40c284d3eadde84d1d7f8f646, State: Running, Role: LEADER
I20260619 16:35:55.795780 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f407d5a40c284d3eadde84d1d7f8f646" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45799 } }
I20260619 16:35:55.796166 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:55.797281 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f407d5a40c284d3eadde84d1d7f8f646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f407d5a40c284d3eadde84d1d7f8f646" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45799 } } }
I20260619 16:35:55.797467 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:55.797449 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f407d5a40c284d3eadde84d1d7f8f646. Latest consensus state: current_term: 1 leader_uuid: "f407d5a40c284d3eadde84d1d7f8f646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f407d5a40c284d3eadde84d1d7f8f646" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45799 } } }
I20260619 16:35:55.797514 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646 [sys.catalog]: This master's current role is: LEADER
W20260619 16:35:55.800742 5594 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:35:55.800827 5594 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:35:55.800900 5588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:55.801651 5588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:55.806191 5588 catalog_manager.cc:1347] Generated new cluster ID: 467033cad2ea4b6390090becf828d977
I20260619 16:35:55.806250 5588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:55.823215 5588 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:55.824067 5588 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:55.827875 5588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f407d5a40c284d3eadde84d1d7f8f646: Generated new TSK 0
I20260619 16:35:55.828485 5588 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:35:55.860152 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 16:35:55.860262 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:35:55.860737 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:55.844016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.62:45799)
0619 16:35:55.844140 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:55.844149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:55.844325 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:35:55.844491 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:55.844493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:55.844546 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:55.844646 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:35:55.844909 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:55.845496 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:55.846755 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:55.847074 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:55.847154 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:55.847517 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:35:55.860409 (+ 12892us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:35:55.860625 (+ 216us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45988: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 16:35:55.860834 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45988: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 16:35:55.862195 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:55.843877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45799 (local address 127.0.0.1:45988)
0619 16:35:55.844221 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:55.844236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:55.844361 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:55.844688 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:55.844691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:55.844699 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:35:55.844809 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:55.844815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:55.845621 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:55.845625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:55.846626 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:55.846642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:55.847183 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:55.847202 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:55.847297 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:55.847346 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:35:55.860635 (+ 13289us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 16:35:55.860864 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45799: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":114,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45799: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42667/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 16:35:55.863224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5507
2026-06-19T16:35:55Z chronyd exiting
I20260619 16:35:55.899652 392 test_util.cc:175] -----------------------------------------------
I20260619 16:35:55.899750 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781886927079793-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 16:35:55.900817 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:35:55.901749 392 webserver.cc:460] Webserver started at https://127.0.0.1:42197/ using document root <none> and password file <none>
I20260619 16:35:55.901876 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:35:55.902134 392 webserver.cc:460] Webserver started at http://127.0.0.1:45317/ using document root <none> and password file <none>
2026-06-19T16:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:55Z Disabled control of system clock
I20260619 16:35:55.910845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38381
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38381
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42197/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/testchainca.cert with env {}
W20260619 16:35:56.016608 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:56.016868 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:56.016907 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:56.020797 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:56.020846 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:56.020861 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:56.020877 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:56.020892 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:35:56.020907 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:35:56.020921 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42197/jwks.json
I20260619 16:35:56.026085 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38381
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42197/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:56.026973 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:56.027851 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:56.036182 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:56.036283 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:56.036278 5610 server_base.cc:1034] running on GCE node
W20260619 16:35:56.036185 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:56.036960 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:56.038010 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:56.039182 5610 hybrid_clock.cc:648] HybridClock initialized: now 1781886956039166 us; error 27 us; skew 500 ppm
I20260619 16:35:56.040880 5610 webserver.cc:460] Webserver started at http://127.0.98.62:36681/ using document root <none> and password file <none>
I20260619 16:35:56.041476 5610 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:56.041539 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:56.041728 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:56.043651 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "9f8cb6f1a06d41edac6d22590e019358"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "b2f2d91f61b9337a195afaf7d9fdaafd"
server_key_iv: "7f8bce6bf73a63f892aa83bf9fa1d6ce"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.044247 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f8cb6f1a06d41edac6d22590e019358"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "b2f2d91f61b9337a195afaf7d9fdaafd"
server_key_iv: "7f8bce6bf73a63f892aa83bf9fa1d6ce"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.047720 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:56.050173 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:56.051364 5610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:35:56.051512 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "9f8cb6f1a06d41edac6d22590e019358"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "b2f2d91f61b9337a195afaf7d9fdaafd"
server_key_iv: "7f8bce6bf73a63f892aa83bf9fa1d6ce"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.051630 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:56.061163 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:56.061785 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:56.061977 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:56.068645 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38381
I20260619 16:35:56.068665 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38381 every 8 connection(s)
I20260619 16:35:56.069538 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:56.072368 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:56.075886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5610
I20260619 16:35:56.076014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:56.076304 392 external_mini_cluster.cc:1428] Setting key 98d8f3354b9319503370d0ddf3d780d7
I20260619 16:35:56.078015 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358: Bootstrap starting.
I20260619 16:35:56.080089 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:56.080775 5677 log.cc:824] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:56.082517 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358: No bootstrap required, opened a new log
I20260619 16:35:56.085760 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:56.077685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.0.98.62:38381)
0619 16:35:56.078357 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:56.078374 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:56.078409 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:35:56.079109 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:56.079126 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:56.079512 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:56.079847 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:56.080321 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.081470 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.082413 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.082982 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.083165 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.084034 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:56.084042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:56.084060 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:56.084434 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:56.084488 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:56.084508 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:56.084622 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:56.084741 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 16:35:56.084994 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260619 16:35:56.085896 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cb6f1a06d41edac6d22590e019358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260619 16:35:56.086153 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:56.086233 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8cb6f1a06d41edac6d22590e019358, State: Initialized, Role: FOLLOWER
I20260619 16:35:56.086628 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cb6f1a06d41edac6d22590e019358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260619 16:35:56.086769 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:56.086840 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:56.086962 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:56.087797 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cb6f1a06d41edac6d22590e019358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260619 16:35:56.088114 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f8cb6f1a06d41edac6d22590e019358; no voters:
I20260619 16:35:56.088402 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:56.088531 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:56.088711 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:35:56.088783 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [term 1 LEADER]: Becoming Leader. State: Replica: 9f8cb6f1a06d41edac6d22590e019358, State: Running, Role: LEADER
I20260619 16:35:56.089064 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cb6f1a06d41edac6d22590e019358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } }
I20260619 16:35:56.089520 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:56.091225 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f8cb6f1a06d41edac6d22590e019358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cb6f1a06d41edac6d22590e019358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } }
I20260619 16:35:56.091274 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8cb6f1a06d41edac6d22590e019358. Latest consensus state: current_term: 1 leader_uuid: "9f8cb6f1a06d41edac6d22590e019358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8cb6f1a06d41edac6d22590e019358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38381 } } }
I20260619 16:35:56.091442 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:56.093572 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358 [sys.catalog]: This master's current role is: LEADER
W20260619 16:35:56.094482 5697 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:35:56.094543 5697 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:35:56.094619 5696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:56.095386 5696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:56.100641 5696 catalog_manager.cc:1347] Generated new cluster ID: 7880dbbfc0c942e5804cba0208fdbebc
I20260619 16:35:56.100708 5696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:56.143682 5696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:56.144686 5696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:56.150380 5696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f8cb6f1a06d41edac6d22590e019358: Generated new TSK 0
I20260619 16:35:56.151081 5696 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 16:35:56.180562 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 16:35:56.180660 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 16:35:56.181223 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:56.165452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38194 (local address 127.0.98.62:38381)
0619 16:35:56.165599 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:56.165604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:56.165749 (+ 145us) server_negotiation.cc:407] Connection header received
0619 16:35:56.165877 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:56.165880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:56.165933 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:56.166047 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:35:56.166250 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.167061 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.167993 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.168345 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.168448 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.168767 (+ 319us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:35:56.180841 (+ 12074us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:35:56.181095 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 16:35:56.181291 5680 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38194: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 16:35:56.181385 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:56.165311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38381 (local address 127.0.0.1:38194)
0619 16:35:56.165661 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:56.165674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:56.165771 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:56.166049 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:56.166051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:56.166058 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:35:56.166149 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:56.166153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.167226 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.167230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:56.167870 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:56.167878 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.168461 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.168479 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:56.168568 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.168590 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:35:56.181032 (+ 12442us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 16:35:56.181171 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38381: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":111,"threads_started":1}
I20260619 16:35:56.181949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5610
2026-06-19T16:35:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 16:35:56.221047 392 mini_oidc.cc:152] Starting JWKS server
I20260619 16:35:56.221956 392 webserver.cc:460] Webserver started at https://127.0.0.1:43981/ using document root <none> and password file <none>
I20260619 16:35:56.222046 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 16:35:56.222298 392 webserver.cc:460] Webserver started at http://127.0.0.1:39447/ using document root <none> and password file <none>
2026-06-19T16:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:56Z Disabled control of system clock
I20260619 16:35:56.230880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42731
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42731
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:35:56.336444 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:56.336710 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:56.336779 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:56.340718 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:56.340799 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:56.340829 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:56.340884 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:56.340924 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:35:56.340948 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:35:56.340968 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43981/jwks.json
I20260619 16:35:56.345599 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42731
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:56.346539 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:56.347795 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:56.355774 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:56.355742 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:56.355767 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:56.355959 5713 server_base.cc:1034] running on GCE node
I20260619 16:35:56.356586 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:56.357658 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:56.358851 5713 hybrid_clock.cc:648] HybridClock initialized: now 1781886956358828 us; error 35 us; skew 500 ppm
I20260619 16:35:56.360790 5713 webserver.cc:460] Webserver started at http://127.0.98.62:46687/ using document root <none> and password file <none>
I20260619 16:35:56.361397 5713 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:56.361485 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:56.361704 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:56.363602 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "1948f2c0be3f44c2aa35f16101519564"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "88cd0676017c2191e3a2ec32c099ee2e"
server_key_iv: "93a60201a0a52727de1098a59f42cff6"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.364156 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "1948f2c0be3f44c2aa35f16101519564"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "88cd0676017c2191e3a2ec32c099ee2e"
server_key_iv: "93a60201a0a52727de1098a59f42cff6"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.367594 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:35:56.369997 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:56.371182 5713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:56.371330 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "1948f2c0be3f44c2aa35f16101519564"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "88cd0676017c2191e3a2ec32c099ee2e"
server_key_iv: "93a60201a0a52727de1098a59f42cff6"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.371440 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:56.387324 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:56.387894 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:56.388080 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:56.394655 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42731
I20260619 16:35:56.394662 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42731 every 8 connection(s)
I20260619 16:35:56.395494 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:56.398466 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:56.403366 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564: Bootstrap starting.
I20260619 16:35:56.405769 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:56.405791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5713
I20260619 16:35:56.405892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:56.406194 392 external_mini_cluster.cc:1428] Setting key a2e72c5c2b560bbbc988c618eab3c404
I20260619 16:35:56.406635 5780 log.cc:824] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:56.408515 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564: No bootstrap required, opened a new log
I20260619 16:35:56.411809 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1948f2c0be3f44c2aa35f16101519564" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42731 } }
I20260619 16:35:56.412004 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:56.412071 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1948f2c0be3f44c2aa35f16101519564, State: Initialized, Role: FOLLOWER
I20260619 16:35:56.412523 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1948f2c0be3f44c2aa35f16101519564" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42731 } }
I20260619 16:35:56.412670 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:56.412756 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:56.412871 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:56.413821 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1948f2c0be3f44c2aa35f16101519564" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42731 } }
I20260619 16:35:56.414183 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1948f2c0be3f44c2aa35f16101519564; no voters:
I20260619 16:35:56.414601 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:56.414535 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:56.414850 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [term 1 LEADER]: Becoming Leader. State: Replica: 1948f2c0be3f44c2aa35f16101519564, State: Running, Role: LEADER
I20260619 16:35:56.415140 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1948f2c0be3f44c2aa35f16101519564" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42731 } }
I20260619 16:35:56.415337 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:56.407245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.0.98.62:42731)
0619 16:35:56.407858 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:56.407874 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:56.407906 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:35:56.408504 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:56.408519 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:56.409010 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:56.409375 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:56.409812 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.410729 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.411614 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.412164 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.412337 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.413200 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:56.413206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:56.413224 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:56.413690 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:56.413742 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:56.413762 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:56.413869 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:56.413985 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:35:56.414238 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":148,"threads_started":1}
I20260619 16:35:56.415800 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:56.416231 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1948f2c0be3f44c2aa35f16101519564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1948f2c0be3f44c2aa35f16101519564" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42731 } } }
I20260619 16:35:56.416337 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:56.416677 5789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:56.418229 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:35:56.419867 5789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:56.418092 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1948f2c0be3f44c2aa35f16101519564. Latest consensus state: current_term: 1 leader_uuid: "1948f2c0be3f44c2aa35f16101519564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1948f2c0be3f44c2aa35f16101519564" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42731 } } }
I20260619 16:35:56.420198 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564 [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:56.425575 5789 catalog_manager.cc:1347] Generated new cluster ID: 6217d7d2b60a459b96abeb82ed68b227
I20260619 16:35:56.425635 5789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:56.453236 5789 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:56.454155 5789 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:56.461143 5789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1948f2c0be3f44c2aa35f16101519564: Generated new TSK 0
I20260619 16:35:56.461797 5789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:35:56.480293 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:56.474877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37304 (local address 127.0.98.62:42731)
0619 16:35:56.475012 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:56.475017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:56.475145 (+ 128us) server_negotiation.cc:407] Connection header received
0619 16:35:56.475279 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:56.475282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:56.475333 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:56.475439 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:35:56.475676 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.476501 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.477650 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.478032 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.478152 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.478480 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:35:56.479917 (+ 1437us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:35:56.480156 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 16:35:56.480362 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37304: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 16:35:56.482160 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:56.474806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42731 (local address 127.0.0.1:37304)
0619 16:35:56.475056 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:56.475068 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:56.475162 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:56.475444 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:56.475446 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:56.475452 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:35:56.475565 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:56.475571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.476651 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.476653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:56.477305 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:56.477365 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.478163 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.478201 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:56.478275 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.478297 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:35:56.480111 (+ 1814us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 16:35:56.481986 (+ 1875us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42731: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43981/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1610,"thread_start_us":101,"threads_started":1}
I20260619 16:35:56.482681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5713
E20260619 16:35:56.491053 5810 webserver.cc:533] Webserver: sslize failed:
E20260619 16:35:56.491135 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T16:35:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T16:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:56Z Disabled control of system clock
I20260619 16:35:56.529703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34035
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34035
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/testchainca.cert with env {}
W20260619 16:35:56.634794 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:56.635027 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:56.635064 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:56.638981 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:56.639027 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:56.639042 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:56.639058 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:56.639096 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 16:35:56.639127 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 16:35:56.639148 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 16:35:56.644263 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:56.645144 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:56.646095 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:56.653981 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:56.654102 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:56.654134 5814 server_base.cc:1034] running on GCE node
W20260619 16:35:56.654206 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:56.655047 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:56.655959 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:56.657138 5814 hybrid_clock.cc:648] HybridClock initialized: now 1781886956657121 us; error 31 us; skew 500 ppm
I20260619 16:35:56.658834 5814 webserver.cc:460] Webserver started at http://127.0.98.62:46695/ using document root <none> and password file <none>
I20260619 16:35:56.659317 5814 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:56.659406 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:56.659646 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:56.661530 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "901d34e70dbd46dbb4648dd738f42d9c"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "0833022e0e5b745e102957ff839ef55f"
server_key_iv: "79df017da0e0147bb1c11190f3d417a6"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.662017 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "901d34e70dbd46dbb4648dd738f42d9c"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "0833022e0e5b745e102957ff839ef55f"
server_key_iv: "79df017da0e0147bb1c11190f3d417a6"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.665477 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:35:56.667804 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:56.669103 5814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:56.669260 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "901d34e70dbd46dbb4648dd738f42d9c"
format_stamp: "Formatted at 2026-06-19 16:35:56 on dist-test-slave-0f28"
server_key: "0833022e0e5b745e102957ff839ef55f"
server_key_iv: "79df017da0e0147bb1c11190f3d417a6"
server_key_version: "encryptionkey@0"
I20260619 16:35:56.669420 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:56.713371 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:56.714097 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:56.714305 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:56.721072 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34035
I20260619 16:35:56.721083 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34035 every 8 connection(s)
I20260619 16:35:56.721993 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:56.724956 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:56.724964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5814
I20260619 16:35:56.725112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:56.725466 392 external_mini_cluster.cc:1428] Setting key 2219280424715e743a037dd5a9b4df75
I20260619 16:35:56.730993 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c: Bootstrap starting.
I20260619 16:35:56.733049 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:56.733877 5881 log.cc:824] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:56.734246 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:56.726679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.0.98.62:34035)
0619 16:35:56.727321 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:56.727343 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:56.727389 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:35:56.728057 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:56.728073 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:56.728448 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:56.728776 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:56.729177 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.730189 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.731045 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.731549 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.731731 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.732564 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:56.732569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:56.732580 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:35:56.732932 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:56.732986 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:35:56.733005 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:56.733121 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:56.733257 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:35:56.733518 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260619 16:35:56.735817 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c: No bootstrap required, opened a new log
I20260619 16:35:56.737035 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:35:56.739879 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d34e70dbd46dbb4648dd738f42d9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260619 16:35:56.740089 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:56.740130 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 901d34e70dbd46dbb4648dd738f42d9c, State: Initialized, Role: FOLLOWER
I20260619 16:35:56.740523 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d34e70dbd46dbb4648dd738f42d9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260619 16:35:56.740658 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:56.740706 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:56.740773 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:56.741621 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d34e70dbd46dbb4648dd738f42d9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260619 16:35:56.741897 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 901d34e70dbd46dbb4648dd738f42d9c; no voters:
I20260619 16:35:56.742133 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:56.742244 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:56.742504 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [term 1 LEADER]: Becoming Leader. State: Replica: 901d34e70dbd46dbb4648dd738f42d9c, State: Running, Role: LEADER
I20260619 16:35:56.742784 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d34e70dbd46dbb4648dd738f42d9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } }
I20260619 16:35:56.743366 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:56.744635 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 901d34e70dbd46dbb4648dd738f42d9c. Latest consensus state: current_term: 1 leader_uuid: "901d34e70dbd46dbb4648dd738f42d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d34e70dbd46dbb4648dd738f42d9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } } }
I20260619 16:35:56.744750 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:56.745049 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "901d34e70dbd46dbb4648dd738f42d9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "901d34e70dbd46dbb4648dd738f42d9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34035 } } }
I20260619 16:35:56.745187 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:56.745213 5895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:56.747848 5895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:56.753196 5895 catalog_manager.cc:1347] Generated new cluster ID: abf023f926ed48e1be5b9fbd7599f80a
I20260619 16:35:56.753259 5895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:56.769816 5895 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:56.770732 5895 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:56.777081 5895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 901d34e70dbd46dbb4648dd738f42d9c: Generated new TSK 0
I20260619 16:35:56.777760 5895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:35:57.252544 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:56.787818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.0.98.62:34035)
0619 16:35:56.787947 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:56.787952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:56.788129 (+ 177us) server_negotiation.cc:407] Connection header received
0619 16:35:56.788276 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:56.788278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:56.788322 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:56.788405 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0619 16:35:56.788660 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.789186 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.790457 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.790690 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.790777 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.791115 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 16:35:57.252176 (+461061us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:35:57.252430 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 16:35:57.252652 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 16:35:57.252802 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:56.787754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34035 (local address 127.0.0.1:56766)
0619 16:35:56.788038 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:56.788053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:56.788166 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:56.788427 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:56.788430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:56.788438 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 16:35:56.788554 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:56.788560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.789292 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.789296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:56.790327 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:56.790337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:56.790796 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:56.790815 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:56.790904 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:56.790930 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 16:35:57.252412 (+461482us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 16:35:57.252598 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34035: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260619 16:35:57.253444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5814
2026-06-19T16:35:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (750 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (131 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:57 dist-test-slave-0f28 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:57 dist-test-slave-0f28 krb5kdc[5922](info): set up 2 sockets
Jun 19 16:35:57 dist-test-slave-0f28 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260619 16:35:58.569628 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:35:58 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:35:58.688287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:58.792724 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:58.792940 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:58.792977 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:58.797053 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:35:58.797106 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:58.797127 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:58.797142 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:35:58.797192 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:35:58.802223 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:58.803119 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:58.804097 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:58.812103 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:58.812160 5935 server_base.cc:1034] running on GCE node
W20260619 16:35:58.812130 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:58.812136 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:58.812865 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:58.813948 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:58.815146 5935 hybrid_clock.cc:648] HybridClock initialized: now 1781886958815124 us; error 43 us; skew 500 ppm
Jun 19 16:35:58 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:58.817936 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:35:58.818737 5935 webserver.cc:460] Webserver started at http://127.0.98.62:35197/ using document root <none> and password file <none>
I20260619 16:35:58.819245 5935 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:58.819301 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:58.819501 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:58.821468 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "51c1721eb10143e7b8e1a98a2b98b75c"
format_stamp: "Formatted at 2026-06-19 16:35:58 on dist-test-slave-0f28"
server_key: "2c823c94c3119fd46d13fdb4d444a8b9"
server_key_iv: "d5833fc91d857ef4569623409521af7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:58.821995 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "51c1721eb10143e7b8e1a98a2b98b75c"
format_stamp: "Formatted at 2026-06-19 16:35:58 on dist-test-slave-0f28"
server_key: "2c823c94c3119fd46d13fdb4d444a8b9"
server_key_iv: "d5833fc91d857ef4569623409521af7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:58.825412 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:58.827606 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:58.828840 5935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:58.828974 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "51c1721eb10143e7b8e1a98a2b98b75c"
format_stamp: "Formatted at 2026-06-19 16:35:58 on dist-test-slave-0f28"
server_key: "2c823c94c3119fd46d13fdb4d444a8b9"
server_key_iv: "d5833fc91d857ef4569623409521af7c"
server_key_version: "encryptionkey@0"
I20260619 16:35:58.829077 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:58.846869 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:58.849712 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:58.849905 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:58.856233 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45391
I20260619 16:35:58.856515 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45391 every 8 connection(s)
I20260619 16:35:58.857151 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:35:58.859692 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:35:58.863263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 5935
I20260619 16:35:58.863397 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:35:58.863727 392 external_mini_cluster.cc:1428] Setting key 06a816bee93bb5fe4739d79efe6e8293
I20260619 16:35:58.866600 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c: Bootstrap starting.
I20260619 16:35:58.869371 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c: Neither blocks nor log segments found. Creating new log.
I20260619 16:35:58.870211 6003 log.cc:824] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:35:58.872419 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c: No bootstrap required, opened a new log
Jun 19 16:35:58 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:58.876391 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260619 16:35:58.876653 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:35:58.876715 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c1721eb10143e7b8e1a98a2b98b75c, State: Initialized, Role: FOLLOWER
I20260619 16:35:58.877161 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260619 16:35:58.877339 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:35:58.877415 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:35:58.877517 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:35:58.878334 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260619 16:35:58.878706 6003 leader_election.cc:302] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51c1721eb10143e7b8e1a98a2b98b75c; no voters:
I20260619 16:35:58.879029 6003 leader_election.cc:288] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:35:58.879094 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:35:58.879284 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [term 1 LEADER]: Becoming Leader. State: Replica: 51c1721eb10143e7b8e1a98a2b98b75c, State: Running, Role: LEADER
I20260619 16:35:58.879616 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } }
I20260619 16:35:58.879796 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:58.865403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.0.98.62:45391)
0619 16:35:58.866009 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:58.866031 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:58.866074 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:35:58.867009 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:58.867033 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:58.867431 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:58.867776 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:58.868635 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:58.870116 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:58.871394 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:58.872053 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:58.872208 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:58.874695 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:58.874703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:58.874723 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:58.874759 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:58.876710 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:58.877192 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:58.877197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:58.877202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:58.877357 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:58.877680 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:58.877683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:58.877685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:58.878061 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:58.878278 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:58.878510 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:35:58.878849 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":138,"threads_started":1}
I20260619 16:35:58.880230 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:35:58.881111 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } } }
I20260619 16:35:58.881278 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [sys.catalog]: This master's current role is: LEADER
I20260619 16:35:58.881443 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51c1721eb10143e7b8e1a98a2b98b75c. Latest consensus state: current_term: 1 leader_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51c1721eb10143e7b8e1a98a2b98b75c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45391 } } }
I20260619 16:35:58.881546 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c [sys.catalog]: This master's current role is: LEADER
W20260619 16:35:58.885793 6024 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:35:58.885885 6024 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:35:58.885958 6012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:35:58.886660 6012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:35:58.891993 6012 catalog_manager.cc:1347] Generated new cluster ID: 733fb85eb91547ebbca34bc436aa2b04
I20260619 16:35:58.892063 6012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:35:58.907706 6012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:35:58.908560 6012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:35:58.915504 6012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51c1721eb10143e7b8e1a98a2b98b75c: Generated new TSK 0
I20260619 16:35:58.916337 6012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:35:58.950150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:59.056074 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:59.056334 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:59.056407 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:59.060388 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:59.060472 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:59.060571 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:35:59.065658 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:59.066596 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:59.067525 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:59.076455 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:59.076464 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:59.076572 6029 server_base.cc:1034] running on GCE node
W20260619 16:35:59.076519 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:59.077131 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:59.077795 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:59.078960 6029 hybrid_clock.cc:648] HybridClock initialized: now 1781886959078941 us; error 29 us; skew 500 ppm
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.081841 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:35:59.082705 6029 webserver.cc:460] Webserver started at http://127.0.98.1:36237/ using document root <none> and password file <none>
I20260619 16:35:59.083226 6029 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:59.083314 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:59.083570 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:59.085561 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "47b488ea675e4418ae9d2de45702cb75"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "be55112d7e1d068102e0377f3b4b4007"
server_key_iv: "77157725f761685089c9d4077c7f2763"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.086118 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "47b488ea675e4418ae9d2de45702cb75"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "be55112d7e1d068102e0377f3b4b4007"
server_key_iv: "77157725f761685089c9d4077c7f2763"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.089748 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:35:59.091971 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.093106 6029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:59.093233 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "47b488ea675e4418ae9d2de45702cb75"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "be55112d7e1d068102e0377f3b4b4007"
server_key_iv: "77157725f761685089c9d4077c7f2763"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.093357 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:59.107178 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:59.110261 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:59.110450 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:59.110982 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:59.111714 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:59.111781 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.111850 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:59.111891 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.120144 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34419
I20260619 16:35:59.120201 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34419 every 8 connection(s)
I20260619 16:35:59.121124 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:35:59.125851 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6029
I20260619 16:35:59.125976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:35:59.126255 392 external_mini_cluster.cc:1428] Setting key 947f3b0754372cab28ca1d5511616a2d
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:59.135747 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.123093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50975 (local address 127.0.98.62:45391)
0619 16:35:59.123240 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.123245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.123959 (+ 714us) server_negotiation.cc:407] Connection header received
0619 16:35:59.125421 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.125425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.125491 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.125573 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.126976 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.127595 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.128730 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.128969 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.129083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.131527 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:59.131534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:59.131540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:59.131572 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:59.133544 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.134052 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.134056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.134059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.134160 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.134548 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.134552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.134554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.134802 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:59.134911 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:59.135382 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 16:35:59.135537 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:35:59.136473 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.123357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45391 (local address 127.0.98.1:50975)
0619 16:35:59.123803 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:59.123845 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:59.125154 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:59.125758 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:59.125765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:59.126214 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:59.126829 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:59.126845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.127705 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.127709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:59.128603 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:59.128612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.129115 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.129119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:59.129286 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.129943 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:59.129959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:59.131354 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:59.133681 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:59.133694 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:59.133705 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:59.133947 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:59.134292 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:59.134295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:59.134296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:59.134437 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:59.134911 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:59.134916 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:59.135141 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 16:35:59.135293 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:35:59.135560 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260619 16:35:59.137709 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45391
I20260619 16:35:59.137964 6158 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:59.138474 6158 heartbeater.cc:511] Master 127.0.98.62:45391 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:35:59.140208 5967 ts_manager.cc:194] Registered new tserver with Master: 47b488ea675e4418ae9d2de45702cb75 (127.0.98.1:34419)
I20260619 16:35:59.141392 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:35:59.154654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:59.261219 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:59.261546 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:59.261622 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:59.265581 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:59.265653 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:59.265748 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:35:59.270753 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:59.271667 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:59.272639 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:59.282070 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:59.282095 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:59.282171 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:59.282476 6163 server_base.cc:1034] running on GCE node
I20260619 16:35:59.282924 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:59.283533 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:59.284732 6163 hybrid_clock.cc:648] HybridClock initialized: now 1781886959284727 us; error 30 us; skew 500 ppm
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.287263 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:35:59.287986 6163 webserver.cc:460] Webserver started at http://127.0.98.2:35241/ using document root <none> and password file <none>
I20260619 16:35:59.288465 6163 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:59.288519 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:59.288669 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:59.290621 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "fae77aa52fb2491eb77e69efd26d72ef"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "0cc16dfda9423df33d3f4c27d070cb48"
server_key_iv: "e432def1d898103cab0a38d2a8727b33"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.291103 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fae77aa52fb2491eb77e69efd26d72ef"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "0cc16dfda9423df33d3f4c27d070cb48"
server_key_iv: "e432def1d898103cab0a38d2a8727b33"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.294914 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:35:59.297286 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.298465 6163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:35:59.298555 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "fae77aa52fb2491eb77e69efd26d72ef"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "0cc16dfda9423df33d3f4c27d070cb48"
server_key_iv: "e432def1d898103cab0a38d2a8727b33"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.298633 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:59.342399 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:59.345086 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:59.345230 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:59.345809 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:59.346541 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:59.346590 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.346628 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:59.346643 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.355441 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41949
I20260619 16:35:59.355515 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41949 every 8 connection(s)
I20260619 16:35:59.356292 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:35:59.360471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6163
I20260619 16:35:59.360625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:35:59.360913 392 external_mini_cluster.cc:1428] Setting key 26eb47d7836817d91715660dfa5ae162
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:35:59.371233 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.358117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45741 (local address 127.0.98.62:45391)
0619 16:35:59.358255 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.358260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.359091 (+ 831us) server_negotiation.cc:407] Connection header received
0619 16:35:59.360381 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.360385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.360445 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.360550 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.361962 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.362593 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.364298 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.364741 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.364860 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.367400 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:59.367407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:59.367412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:59.367446 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:59.369151 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.369709 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.369715 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.369718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.369799 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.370128 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.370134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.370137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.370389 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:59.370507 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:59.370950 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 16:35:59.371097 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:35:59.372084 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.358521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45391 (local address 127.0.98.2:45741)
0619 16:35:59.358946 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:59.358987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:59.360159 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:59.360805 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:59.360811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:59.361204 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:59.361805 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:59.361819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.362722 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.362727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:59.364180 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:59.364189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.364889 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.364892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:59.365023 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.365680 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:59.365702 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:59.367231 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:59.369278 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:59.369283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:59.369294 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:59.369606 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:59.369892 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:59.369894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:59.369895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:59.370024 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:59.370489 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:59.370493 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:59.370704 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 16:35:59.370887 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:35:59.371202 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
I20260619 16:35:59.373260 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:35:59.373551 6292 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:59.374019 6292 heartbeater.cc:511] Master 127.0.98.62:45391 requested a full tablet report, sending...
I20260619 16:35:59.375123 5967 ts_manager.cc:194] Registered new tserver with Master: fae77aa52fb2491eb77e69efd26d72ef (127.0.98.2:41949)
I20260619 16:35:59.375946 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:35:59.388659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:35:59.493992 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:35:59.494217 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:35:59.494257 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:35:59.498247 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:35:59.498296 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:35:59.498350 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:35:59.503392 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:35:59.504258 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:35:59.505153 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:35:59.513931 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:59.513962 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:35:59.513993 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:35:59.514175 6297 server_base.cc:1034] running on GCE node
I20260619 16:35:59.514537 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:35:59.515101 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:35:59.516266 6297 hybrid_clock.cc:648] HybridClock initialized: now 1781886959516240 us; error 47 us; skew 500 ppm
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.518906 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:35:59.519721 6297 webserver.cc:460] Webserver started at http://127.0.98.3:39829/ using document root <none> and password file <none>
I20260619 16:35:59.520236 6297 fs_manager.cc:359] Metadata directory not provided
I20260619 16:35:59.520323 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:35:59.520541 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:35:59.522464 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "bbc633a35bb0433a9b8402d528279c5f"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "29d04076fa086408ecd14ca2f4feac64"
server_key_iv: "8bed498c2deba57634809ea30e2edb65"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.522993 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bbc633a35bb0433a9b8402d528279c5f"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "29d04076fa086408ecd14ca2f4feac64"
server_key_iv: "8bed498c2deba57634809ea30e2edb65"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.526422 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 16:35:59.528589 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.529749 6297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:35:59.529875 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "bbc633a35bb0433a9b8402d528279c5f"
format_stamp: "Formatted at 2026-06-19 16:35:59 on dist-test-slave-0f28"
server_key: "29d04076fa086408ecd14ca2f4feac64"
server_key_iv: "8bed498c2deba57634809ea30e2edb65"
server_key_version: "encryptionkey@0"
I20260619 16:35:59.529989 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:35:59.553362 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:35:59.556089 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:35:59.556290 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:35:59.556836 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:35:59.557614 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:35:59.557686 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.557749 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:35:59.557801 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:35:59.566413 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35905
I20260619 16:35:59.566444 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35905 every 8 connection(s)
I20260619 16:35:59.567386 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:35:59.574107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6297
I20260619 16:35:59.574244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:35:59.574512 392 external_mini_cluster.cc:1428] Setting key 03fa6a5cd0224e22c6fb6688ded4864e
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:35:59.580941 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.569194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48621 (local address 127.0.98.62:45391)
0619 16:35:59.569347 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.569351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.570280 (+ 929us) server_negotiation.cc:407] Connection header received
0619 16:35:59.571619 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.571622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.571681 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.571763 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.572843 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.573451 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.574956 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.575196 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.575307 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.577769 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:59.577773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:59.577776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:59.577798 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:59.578969 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.579450 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.579454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.579456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.579534 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.579881 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.579886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.579888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.580107 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:59.580217 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:59.580665 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 16:35:59.580776 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:35:59.581830 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.569687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45391 (local address 127.0.98.3:48621)
0619 16:35:59.570126 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:35:59.570167 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:35:59.571399 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:35:59.571934 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:35:59.571941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:35:59.572273 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:35:59.572701 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:59.572712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.573585 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.573588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:59.574835 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:35:59.574843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.575336 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.575339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:35:59.575549 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.576167 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:35:59.576183 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:35:59.577620 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:35:59.579080 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:59.579086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:59.579106 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:59.579345 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:59.579641 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:35:59.579644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:35:59.579646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:35:59.579776 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:35:59.580214 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:35:59.580220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:35:59.580441 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 16:35:59.580623 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 16:35:59.580902 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":126,"threads_started":1}
I20260619 16:35:59.582819 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45391
I20260619 16:35:59.583019 6426 heartbeater.cc:464] Registering TS with master...
I20260619 16:35:59.583415 6426 heartbeater.cc:511] Master 127.0.98.62:45391 requested a full tablet report, sending...
I20260619 16:35:59.584357 5967 ts_manager.cc:194] Registered new tserver with Master: bbc633a35bb0433a9b8402d528279c5f (127.0.98.3:35905)
I20260619 16:35:59.584981 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48621
I20260619 16:35:59.588130 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/test-admin.keytab.
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.603886 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 16:35:59.606557 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:59.605115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.0.98.1:34419)
0619 16:35:59.605526 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.605541 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.605564 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:35:59.605619 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.605624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.605797 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.605917 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.606371 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47764: BlockingRecv error: recv got EOF from 127.0.0.1:47764 (error 108)
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
W20260619 16:35:59.606643 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.607749 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.608669 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:59.618126 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.609591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.0.98.1:34419)
0619 16:35:59.609716 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.609723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.609768 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:35:59.609924 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.609934 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.609972 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.610060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.610584 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.611346 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.612215 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.612575 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.612696 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.614956 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:59.614964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:59.614975 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:59.614998 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:59.616471 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.616893 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.616897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.616898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.616970 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.617231 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.617233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.617234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.617502 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:59.617647 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:59.617871 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 16:35:59.617987 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 16:35:59.621726 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:59.620731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.0.98.1:34419)
0619 16:35:59.620846 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.620849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.620944 (+ 95us) server_negotiation.cc:407] Connection header received
0619 16:35:59.621052 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.621054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.621096 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.621180 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.621618 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47782: BlockingRecv error: recv got EOF from 127.0.0.1:47782 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 16:35:59.621896 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.622828 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.623631 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:59.631349 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.624302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.0.98.1:34419)
0619 16:35:59.624404 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.624407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.624615 (+ 208us) server_negotiation.cc:407] Connection header received
0619 16:35:59.624653 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.624655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.624689 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.624761 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.625222 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.625758 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.626522 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.626733 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.626809 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.628761 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:59.628764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:59.628765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:59.628786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:59.630111 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.630479 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.630481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.630482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.630551 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.630773 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.630775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.630776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.630908 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:59.630995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:59.631174 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:35:59.631254 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 16:35:59.633895 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:35:59.632969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.0.98.1:34419)
0619 16:35:59.633084 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.633090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.633128 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:35:59.633264 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.633267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.633373 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.633457 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.633810 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47800: BlockingRecv error: recv got EOF from 127.0.0.1:47800 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 16:35:59.634253 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781886927079793-392-0/krb5kdc/krb5cc)
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.635121 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:35:59.635923 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:35:59.644016 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:35:59.636663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.0.98.1:34419)
0619 16:35:59.636794 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:35:59.636797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:35:59.636810 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:35:59.636960 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:35:59.636967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:35:59.637001 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:35:59.637076 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:35:59.637636 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.638085 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.638877 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:35:59.639080 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:35:59.639157 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:35:59.641175 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:35:59.641177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:35:59.641179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:35:59.641198 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:35:59.642579 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.642961 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.642963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.642964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.643027 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:35:59.643260 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:35:59.643262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:35:59.643263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:35:59.643388 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:35:59.643473 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:35:59.643751 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 16:35:59.643859 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 16:35:59.645202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6029
I20260619 16:35:59.651676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6163
I20260619 16:35:59.656884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6297
I20260619 16:35:59.661942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 5935
2026-06-19T16:35:59Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2161 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[6446](info): set up 2 sockets
Jun 19 16:35:59 dist-test-slave-0f28 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260619 16:36:00.705760 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:36:00 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 16:36:00.826027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36897
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:00.935987 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:00.936237 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:00.936276 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:00.940265 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:00.940318 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:00.940337 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:00.940353 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:00.940397 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:00.945604 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:00.946568 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:00.947559 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:00.955790 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:00.955812 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:00.955802 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:00.956046 6459 server_base.cc:1034] running on GCE node
I20260619 16:36:00.956637 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:00.957727 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:00.958911 6459 hybrid_clock.cc:648] HybridClock initialized: now 1781886960958881 us; error 41 us; skew 500 ppm
Jun 19 16:36:00 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:00.961671 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 16:36:00.962785 6459 webserver.cc:460] Webserver started at http://127.0.98.62:43915/ using document root <none> and password file <none>
I20260619 16:36:00.963469 6459 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:00.963555 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:00.963758 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:00.965780 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "47d0f9f98d954ad985e316c65f7a5cac"
format_stamp: "Formatted at 2026-06-19 16:36:00 on dist-test-slave-0f28"
server_key: "1d1f3fb7bd5f64dae5da8e6fe44dd9e3"
server_key_iv: "c72c7ffc945da02486630414908caeff"
server_key_version: "encryptionkey@0"
I20260619 16:36:00.966322 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "47d0f9f98d954ad985e316c65f7a5cac"
format_stamp: "Formatted at 2026-06-19 16:36:00 on dist-test-slave-0f28"
server_key: "1d1f3fb7bd5f64dae5da8e6fe44dd9e3"
server_key_iv: "c72c7ffc945da02486630414908caeff"
server_key_version: "encryptionkey@0"
I20260619 16:36:00.970038 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:36:00.972337 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:00.973628 6459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:00.973757 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "47d0f9f98d954ad985e316c65f7a5cac"
format_stamp: "Formatted at 2026-06-19 16:36:00 on dist-test-slave-0f28"
server_key: "1d1f3fb7bd5f64dae5da8e6fe44dd9e3"
server_key_iv: "c72c7ffc945da02486630414908caeff"
server_key_version: "encryptionkey@0"
I20260619 16:36:00.973863 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:00.990811 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:00.993762 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:00.993958 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:01.000511 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36897
I20260619 16:36:01.000815 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36897 every 8 connection(s)
I20260619 16:36:01.001461 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:01.002220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6459
I20260619 16:36:01.002342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:36:01.002668 392 external_mini_cluster.cc:1428] Setting key 3735159d97754ef0cff0a445ce67f3c9
I20260619 16:36:01.004716 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:01.010057 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Bootstrap starting.
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:01.012342 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:01.013053 6527 log.cc:824] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:01.015233 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: No bootstrap required, opened a new log
I20260619 16:36:01.016036 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.003977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.0.98.62:36897)
0619 16:36:01.004395 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.004409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.004441 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:36:01.005132 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.005148 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.005513 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.005839 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.006588 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.007545 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.008390 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.008839 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.008997 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.011472 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.011477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.011489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.011513 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.013248 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.013795 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.013800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.013804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.013910 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.014204 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.014207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.014208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.014583 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.014738 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.015012 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.015325 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":135,"threads_started":1}
I20260619 16:36:01.018725 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } }
I20260619 16:36:01.018953 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:01.019032 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d0f9f98d954ad985e316c65f7a5cac, State: Initialized, Role: FOLLOWER
I20260619 16:36:01.019466 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } }
I20260619 16:36:01.019619 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:01.019706 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:01.019800 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:01.020630 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } }
I20260619 16:36:01.020949 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d0f9f98d954ad985e316c65f7a5cac; no voters:
I20260619 16:36:01.021256 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:36:01.021400 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:01.021653 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 LEADER]: Becoming Leader. State: Replica: 47d0f9f98d954ad985e316c65f7a5cac, State: Running, Role: LEADER
I20260619 16:36:01.021907 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } }
I20260619 16:36:01.022466 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:01.023484 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } }
I20260619 16:36:01.023607 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:01.023963 6541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:01.024377 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d0f9f98d954ad985e316c65f7a5cac. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } }
I20260619 16:36:01.024497 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:01.026614 6541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:01.031534 6541 catalog_manager.cc:1347] Generated new cluster ID: b9b51045b86c49e285eaf80ec741b0ab
I20260619 16:36:01.031599 6541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:01.053104 6541 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:36:01.054113 6541 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:01.060071 6541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Generated new TSK 0
I20260619 16:36:01.060691 6541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 16:36:01.097124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36897
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:01.209133 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:01.209494 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:01.209587 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:01.213542 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:01.213636 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:01.213724 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:01.218891 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:01.219928 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:01.220919 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:01.230348 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.230363 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.230337 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:01.230748 6553 server_base.cc:1034] running on GCE node
I20260619 16:36:01.231142 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:01.231745 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:01.232945 6553 hybrid_clock.cc:648] HybridClock initialized: now 1781886961232933 us; error 24 us; skew 500 ppm
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:01.236079 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:36:01.236894 6553 webserver.cc:460] Webserver started at http://127.0.98.1:34647/ using document root <none> and password file <none>
I20260619 16:36:01.237494 6553 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:01.237562 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:01.237761 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:01.239694 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "419a6fb8404345bb876dafd02d76affb"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "21361318235c6c77160f91005c59fcc6"
server_key_iv: "ab151247d047cdcd1c3be88c0f4e6f87"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.240239 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "419a6fb8404345bb876dafd02d76affb"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "21361318235c6c77160f91005c59fcc6"
server_key_iv: "ab151247d047cdcd1c3be88c0f4e6f87"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.243932 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 16:36:01.246438 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.247718 6553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:36:01.247838 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "419a6fb8404345bb876dafd02d76affb"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "21361318235c6c77160f91005c59fcc6"
server_key_iv: "ab151247d047cdcd1c3be88c0f4e6f87"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.247969 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:01.265913 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:01.268577 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:01.268776 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:01.269331 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:01.270221 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:01.270300 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.270382 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:01.270447 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.279505 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37615
I20260619 16:36:01.279528 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37615 every 8 connection(s)
I20260619 16:36:01.280367 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:01.283711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6553
I20260619 16:36:01.283800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:36:01.284076 392 external_mini_cluster.cc:1428] Setting key 0b1c39320976465d3c25bb2a7673d6ec
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:36:01.295921 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.282131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48627 (local address 127.0.98.62:36897)
0619 16:36:01.282271 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.282276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.283220 (+ 944us) server_negotiation.cc:407] Connection header received
0619 16:36:01.284590 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.284594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.284650 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.284718 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.286015 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.286860 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.288685 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.289093 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.289203 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.292202 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.292206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.292209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.292234 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.293839 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.294342 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.294347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.294349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.294437 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.294849 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.294854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.294856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.295042 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.295150 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.295688 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.295793 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 16:36:01.296823 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.282642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:48627)
0619 16:36:01.283071 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.283112 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.284381 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.284903 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.284910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.285358 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.285870 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.285884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.286992 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.287017 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.288553 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.288567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.289310 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.289334 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.289609 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.290290 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.290317 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.292040 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.293967 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.293973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.293987 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.294240 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.294537 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.294540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.294542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.294739 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.295147 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.295153 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.295393 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.295599 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.295863 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260619 16:36:01.298035 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36897
I20260619 16:36:01.298259 6682 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:01.298713 6682 heartbeater.cc:511] Master 127.0.98.62:36897 requested a full tablet report, sending...
I20260619 16:36:01.300256 6491 ts_manager.cc:194] Registered new tserver with Master: 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615)
I20260619 16:36:01.301971 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 16:36:01.312656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36897
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:01.419445 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:01.419710 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:01.419754 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:01.424011 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:01.424108 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:01.424199 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:01.429986 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:01.431038 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:01.432065 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:01.441555 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.441531 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.441591 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:01.441859 6687 server_base.cc:1034] running on GCE node
I20260619 16:36:01.442219 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:01.442824 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:01.444015 6687 hybrid_clock.cc:648] HybridClock initialized: now 1781886961443978 us; error 46 us; skew 500 ppm
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:01.446775 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:36:01.447618 6687 webserver.cc:460] Webserver started at http://127.0.98.2:38517/ using document root <none> and password file <none>
I20260619 16:36:01.448173 6687 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:01.448266 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:01.448499 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:01.450515 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "bdd289fed93242188e10f130ae14e046"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "d5e1b5efad26e7baa0c30c0f0a24fbca"
server_key_iv: "53c399c1beec7cf6de0ac4b51b727fc5"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.451104 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bdd289fed93242188e10f130ae14e046"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "d5e1b5efad26e7baa0c30c0f0a24fbca"
server_key_iv: "53c399c1beec7cf6de0ac4b51b727fc5"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.454692 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:36:01.457111 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.458416 6687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:01.458604 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "bdd289fed93242188e10f130ae14e046"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "d5e1b5efad26e7baa0c30c0f0a24fbca"
server_key_iv: "53c399c1beec7cf6de0ac4b51b727fc5"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.458715 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:01.468046 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:01.471071 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:01.471269 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:01.471822 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:01.472601 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:01.472672 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.472736 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:01.472790 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.482069 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39291
I20260619 16:36:01.482091 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39291 every 8 connection(s)
I20260619 16:36:01.482936 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:01.488401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6687
I20260619 16:36:01.488544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:36:01.488838 392 external_mini_cluster.cc:1428] Setting key ffcb9fc5870ccd908ae92625200ed1e0
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:01.497231 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.484700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39357 (local address 127.0.98.62:36897)
0619 16:36:01.484828 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.484832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.485764 (+ 932us) server_negotiation.cc:407] Connection header received
0619 16:36:01.487026 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.487029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.487075 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.487150 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.488337 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.489040 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.490460 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.490672 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.490755 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.493371 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.493378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.493384 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.493435 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.495178 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.495689 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.495695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.495699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.495795 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.496195 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.496200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.496207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.496374 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.496495 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.496953 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.497067 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:36:01.498029 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.485080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:39357)
0619 16:36:01.485614 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.485655 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.486817 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.487305 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.487311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.487695 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.488175 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.488193 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.489165 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.489168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.490322 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.490330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.490819 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.490823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.491046 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.491648 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.491672 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.493190 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.495305 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.495310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.495320 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.495584 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.495914 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.495917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.495918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.496075 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.496491 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.496497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.496711 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.496886 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.497178 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:36:01.499157 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36897
I20260619 16:36:01.499385 6816 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:01.499861 6816 heartbeater.cc:511] Master 127.0.98.62:36897 requested a full tablet report, sending...
I20260619 16:36:01.500851 6491 ts_manager.cc:194] Registered new tserver with Master: bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 16:36:01.501639 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:36:01.517092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36897
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:01.625857 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:01.626094 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:01.626134 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:01.630098 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:01.630149 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:01.630239 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:01.635219 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:01.636159 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:01.637109 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:01.646468 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.646526 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.646526 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:01.646728 6821 server_base.cc:1034] running on GCE node
I20260619 16:36:01.647107 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:01.647805 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:01.648995 6821 hybrid_clock.cc:648] HybridClock initialized: now 1781886961648976 us; error 31 us; skew 500 ppm
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:01.651829 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:36:01.652767 6821 webserver.cc:460] Webserver started at http://127.0.98.3:46163/ using document root <none> and password file <none>
I20260619 16:36:01.653398 6821 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:01.653497 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:01.653746 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:01.655707 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a982f53b199477faf9d8df3904c89a2"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "658efb6ad19116ca20147b8e04ab5118"
server_key_iv: "e249dc5334b4f3ac6ce1f038fb421893"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.656263 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a982f53b199477faf9d8df3904c89a2"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "658efb6ad19116ca20147b8e04ab5118"
server_key_iv: "e249dc5334b4f3ac6ce1f038fb421893"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.659826 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:36:01.662329 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.663530 6821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:36:01.663666 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "1a982f53b199477faf9d8df3904c89a2"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "658efb6ad19116ca20147b8e04ab5118"
server_key_iv: "e249dc5334b4f3ac6ce1f038fb421893"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.663781 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:01.678826 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:01.681573 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:01.681773 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:01.682292 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:01.683035 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:01.683080 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.683145 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:01.683178 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.691560 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45481
I20260619 16:36:01.691581 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45481 every 8 connection(s)
I20260619 16:36:01.692431 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:01.692830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6821
I20260619 16:36:01.692945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:36:01.693224 392 external_mini_cluster.cc:1428] Setting key 4fa4d140fbbb3ce00a3e51a42e817b32
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:01.705056 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.694204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45003 (local address 127.0.98.62:36897)
0619 16:36:01.695718 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.695721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.695734 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:01.696684 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.696686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.696736 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.696804 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.697953 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.698501 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.699568 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.699773 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.699849 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.702094 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.702097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.702099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.702122 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.703398 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.703844 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.703847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.703848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.703919 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.704197 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.704199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.704200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.704357 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.704430 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.704858 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.704946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1431}
I20260619 16:36:01.705824 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.694736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:45003)
0619 16:36:01.695284 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.695338 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.696506 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.696960 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.696967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.697292 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.697804 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.697817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.698610 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.698612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.699435 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.699443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.699888 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.699890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.700020 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.700564 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.700578 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.701950 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.703517 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.703521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.703531 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.703762 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.704001 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.704003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.704004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.704117 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.704440 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.704444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.704666 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.704810 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.705025 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":148,"threads_started":1}
I20260619 16:36:01.706948 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36897
I20260619 16:36:01.707182 6950 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:01.707572 6950 heartbeater.cc:511] Master 127.0.98.62:36897 requested a full tablet report, sending...
I20260619 16:36:01.708467 6491 ts_manager.cc:194] Registered new tserver with Master: 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3:45481)
I20260619 16:36:01.709033 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45003
I20260619 16:36:01.718700 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 16:36:01.744895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37975
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36897,127.0.98.61:37975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:01.852914 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:01.853137 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:01.853175 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:01.857101 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:01.857152 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:01.857172 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:01.857188 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:01.857240 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:01.862466 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36897,127.0.98.61:37975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
W20260619 16:36:01.862655 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:36897,127.0.98.61:37975'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 16:36:01.863462 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:01.864390 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:01.872429 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:01.872373 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:01.872512 6955 server_base.cc:1034] running on GCE node
W20260619 16:36:01.872665 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:01.873140 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:01.874199 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:01.875352 6955 hybrid_clock.cc:648] HybridClock initialized: now 1781886961875337 us; error 30 us; skew 500 ppm
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:01.877877 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 16:36:01.878729 6955 webserver.cc:460] Webserver started at http://127.0.98.61:35041/ using document root <none> and password file <none>
I20260619 16:36:01.879194 6955 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:01.879247 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:01.879439 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:01.881291 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/instance:
uuid: "5e1fed4b2b9d497ebe98f682df9493da"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "adac008b6cb5d1e661a371938dfb4612"
server_key_iv: "4fbeef54b8917dbfae95369193bc8d3f"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.881891 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal/instance:
uuid: "5e1fed4b2b9d497ebe98f682df9493da"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "adac008b6cb5d1e661a371938dfb4612"
server_key_iv: "4fbeef54b8917dbfae95369193bc8d3f"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.885406 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:36:01.887632 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:01.888700 6955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:01.888826 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
uuid: "5e1fed4b2b9d497ebe98f682df9493da"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "adac008b6cb5d1e661a371938dfb4612"
server_key_iv: "4fbeef54b8917dbfae95369193bc8d3f"
server_key_version: "encryptionkey@0"
I20260619 16:36:01.888921 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:01.901893 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:01.904492 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:01.904685 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:01.911334 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37975
I20260619 16:36:01.911631 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37975 every 8 connection(s)
I20260619 16:36:01.912215 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:36:01.914870 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:01.918392 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:36:01.920258 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 6955
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:01.931932 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.920268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.0.98.62:36897)
0619 16:36:01.920486 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.920493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.921201 (+ 708us) server_negotiation.cc:407] Connection header received
0619 16:36:01.922219 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.922222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.922277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.922377 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.923270 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.923904 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.924980 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.925192 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.925303 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.927891 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.927895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.927898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.927921 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.929645 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.930266 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.930272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.930277 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.930368 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.930790 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.930793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.930795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.931062 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.931140 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.931630 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.931746 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:36:01.932709 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.920623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:60810)
0619 16:36:01.921058 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.921107 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.922021 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.922422 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.922431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.922857 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.923139 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.923158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.924028 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.924031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.924845 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.924855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.925355 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.925360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.925629 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.926228 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.926251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.927753 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.929773 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.929780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.929787 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.930165 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.930457 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.930460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.930462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.930673 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.931180 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.931189 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.931406 (+ 217us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.931548 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.931864 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":79,"thread_start_us":129,"threads_started":1}
I20260619 16:36:01.932713 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.921074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.98.61:37975)
0619 16:36:01.921492 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.921505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.921543 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:36:01.922050 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.922074 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.922258 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.922505 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.923373 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.924412 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.925206 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.925672 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.925786 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.928363 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.928372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.928385 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.928410 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.930185 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.930758 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.930765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.930768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.930862 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.931202 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.931206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.931207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.931433 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.931698 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.931904 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.932046 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":315,"thread_start_us":109,"threads_started":1}
I20260619 16:36:01.934053 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:01.941157 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.934449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:41754)
0619 16:36:01.934556 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.934568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.934636 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.934852 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.934856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.935018 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.935097 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.935102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.935750 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.935752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.936222 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.936228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.936671 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.936674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.936749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.937111 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.937122 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.938337 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.939947 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.939950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.939951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.940128 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.940356 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.940358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.940359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.940426 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.940741 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.940744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.940814 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.940914 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.941027 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:36:01.941237 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.934534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.0.98.61:37975)
0619 16:36:01.934655 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.934663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.934677 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:01.934712 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.934715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.934751 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.934821 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.935192 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.935663 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.936311 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.936546 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.936620 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.938452 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.938455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.938457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.938476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.939838 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.940213 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.940215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.940216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.940279 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.940498 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.940500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.940501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.940636 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.940706 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.940963 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.941082 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":34}
I20260619 16:36:01.943993 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Bootstrap starting.
I20260619 16:36:01.946141 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:01.946822 7023 log.cc:824] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:01.948485 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: No bootstrap required, opened a new log
I20260619 16:36:01.951500 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } }
I20260619 16:36:01.951712 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:01.951803 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1fed4b2b9d497ebe98f682df9493da, State: Initialized, Role: FOLLOWER
I20260619 16:36:01.952229 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } }
I20260619 16:36:01.952685 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } } }
I20260619 16:36:01.952785 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:01.953365 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:36:01.957728 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:36:01.957814 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:36:01.959393 6955 master_runner.cc:428] Detected that this master 5e1fed4b2b9d497ebe98f682df9493da is joining an existing cluster
I20260619 16:36:01.959450 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:37975
I20260619 16:36:01.970220 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.961427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.0.98.62:36897)
0619 16:36:01.961543 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.961548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.961962 (+ 414us) server_negotiation.cc:407] Connection header received
0619 16:36:01.962214 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.962218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.962300 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.962363 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.963045 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.963908 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.965004 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.965234 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.965366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.966527 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.966533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.966536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.966562 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.968534 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.968970 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.968974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.968976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.969062 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.969355 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.969362 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.969366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.969544 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.969653 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.969956 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.970070 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:36:01.970697 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.961383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:60826)
0619 16:36:01.961856 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.961872 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.962040 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.962544 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.962548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.962798 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.962932 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.962938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.964008 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.964011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.964896 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.964902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.965388 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.965392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.965539 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.966049 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.966061 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.966355 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.968625 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.968631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.968633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.968859 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.969161 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.969164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.969166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.969246 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.969642 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.969645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.969757 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.969880 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.970023 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260619 16:36:01.970954 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.961484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.98.61:37975)
0619 16:36:01.962769 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.962774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.963267 (+ 493us) server_negotiation.cc:407] Connection header received
0619 16:36:01.963390 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.963396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.963437 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.963520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.963926 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.964678 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.965896 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.966213 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.966284 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.967544 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.967550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.967554 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.967583 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.969265 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.969757 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.969760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.969762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.969852 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.970122 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.970125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.970127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.970294 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.970375 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.970653 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.970759 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"spinlock_wait_cycles":25344}
I20260619 16:36:01.971386 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.961686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:41758)
0619 16:36:01.963183 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.963202 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.963299 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.963543 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.963546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.963730 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.963812 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.963817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.964794 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.964797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.965772 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.965782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.966408 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.966411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.966509 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.966978 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.966993 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.967371 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.969415 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.969417 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.969419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.969666 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.969956 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.969959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.969960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.970035 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.970430 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.970432 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.970499 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.971112 (+ 613us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.971224 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1401,"mutex_wait_us":23,"thread_start_us":149,"threads_started":1}
Jun 19 16:36:01 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:01.983635 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.974954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41774 (local address 127.0.98.61:37975)
0619 16:36:01.975061 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:01.975065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:01.975336 (+ 271us) server_negotiation.cc:407] Connection header received
0619 16:36:01.975512 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:01.975515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:01.975560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:01.975640 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:01.976195 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.976961 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.977968 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.978290 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.978393 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.980424 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:01.980427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:01.980430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:01.980453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:01.981911 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.982401 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.982407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.982409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.982485 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:01.982773 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:01.982777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:01.982779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:01.982972 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:01.983068 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:01.983392 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 16:36:01.983492 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:36:01.983691 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:01.974908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:41774)
0619 16:36:01.975215 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:01.975261 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:01.975384 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:01.975674 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:01.975679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:01.975993 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:01.976093 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.976098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.977050 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.977053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.977831 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:01.977840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:01.978393 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:01.978395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:01.978482 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:01.978987 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:01.978999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:01.980303 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:01.982040 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.982050 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.982056 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.982301 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.982589 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:01.982591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:01.982593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:01.982684 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:01.983082 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:01.983090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:01.983201 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 16:36:01.983343 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:36:01.983520 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":94,"threads_started":1}
I20260619 16:36:01.985589 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:37975: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 37975
}
attrs {
promote: true
}
}
dest_uuid: "47d0f9f98d954ad985e316c65f7a5cac"
cas_config_opid_index: -1
I20260619 16:36:01.986297 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } }
I20260619 16:36:01.988238 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } }
I20260619 16:36:01.989392 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:01.990074 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:37975
I20260619 16:36:01.990490 6955 master.cc:537] Master@127.0.98.61:37975 shutting down...
I20260619 16:36:01.991060 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Refusing update from remote peer 47d0f9f98d954ad985e316c65f7a5cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 16:36:01.991060 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } }
I20260619 16:36:01.991163 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:01.991930 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:36:01.994205 6475 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 16:36:01.995126 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 16:36:01.995147 6475 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac -> Peer 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61:37975): Couldn't send request to peer 5e1fed4b2b9d497ebe98f682df9493da. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:36:01.995402 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:01.995530 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 16:36:02.010013 6955 master.cc:552] Master@127.0.98.61:37975 shutdown complete.
I20260619 16:36:02.010071 6955 master_runner.cc:305] Clearing existing system tablet
I20260619 16:36:02.010314 6955 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.010363 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.012319 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:02.013242 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.013522 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.013643 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 16:36:02.013681 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
uuid: "5e1fed4b2b9d497ebe98f682df9493da"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "adac008b6cb5d1e661a371938dfb4612"
server_key_iv: "4fbeef54b8917dbfae95369193bc8d3f"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.015803 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 16:36:02.018568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33025
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:36:02.018679 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:36:02.018754 6955 log.cc:1196] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 16:36:02.018982 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Deleting consensus metadata
I20260619 16:36:02.019145 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:36897
I20260619 16:36:02.020246 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36897
I20260619 16:36:02.029037 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.020925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.0.98.62:36897)
0619 16:36:02.021040 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.021045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.021303 (+ 258us) server_negotiation.cc:407] Connection header received
0619 16:36:02.021455 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.021458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.021506 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.021604 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.022039 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.022590 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.023609 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.023928 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.024044 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.025697 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.025701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.025703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.025728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.027350 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.027856 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.027860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.027862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.027948 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.028233 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.028238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.028240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.028420 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.028525 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.028781 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.028891 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:36:02.029053 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.020855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:60832)
0619 16:36:02.021194 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.021216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.021367 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.021624 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.021626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.021824 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.021924 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.021929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.022725 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.022730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.023469 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.023480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.024047 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.024051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.024153 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.025137 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.025149 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.025557 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.027456 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.027461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.027463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.027733 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.028047 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.028050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.028052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.028143 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.028513 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.028516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.028616 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.028726 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.028867 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260619 16:36:02.029873 6511 tablet_copy_service.cc:140] P 47d0f9f98d954ad985e316c65f7a5cac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5e1fed4b2b9d497ebe98f682df9493da ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60832)
I20260619 16:36:02.030043 6511 tablet_copy_service.cc:161] P 47d0f9f98d954ad985e316c65f7a5cac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5e1fed4b2b9d497ebe98f682df9493da at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60832: session id = 5e1fed4b2b9d497ebe98f682df9493da-00000000000000000000000000000000
I20260619 16:36:02.031358 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:36:02.032925 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:36:02.035211 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: tablet copy: Starting download of 0 data blocks...
I20260619 16:36:02.035387 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: tablet copy: Starting download of 1 WAL segments...
I20260619 16:36:02.037120 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:36:02.038772 6511 tablet_copy_service.cc:342] P 47d0f9f98d954ad985e316c65f7a5cac: Request end of tablet copy session 5e1fed4b2b9d497ebe98f682df9493da-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60832
I20260619 16:36:02.038882 6511 tablet_copy_service.cc:433] P 47d0f9f98d954ad985e316c65f7a5cac: ending tablet copy session 5e1fed4b2b9d497ebe98f682df9493da-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5e1fed4b2b9d497ebe98f682df9493da
I20260619 16:36:02.046077 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:02.047814 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.050431 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.050556 6955 server_base.cc:1034] running on GCE node
W20260619 16:36:02.050707 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.050993 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:02.051312 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:02.052469 6955 hybrid_clock.cc:648] HybridClock initialized: now 1781886962052441 us; error 42 us; skew 500 ppm
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:02.054068 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 16:36:02.054524 6955 webserver.cc:460] Webserver started at http://127.0.98.61:42639/ using document root <none> and password file <none>
I20260619 16:36:02.054711 6955 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.054754 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.055998 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:36:02.056967 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.057209 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 16:36:02.057277 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
uuid: "5e1fed4b2b9d497ebe98f682df9493da"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "adac008b6cb5d1e661a371938dfb4612"
server_key_iv: "4fbeef54b8917dbfae95369193bc8d3f"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.057369 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.078706 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:02.080791 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:02.085332 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37975
I20260619 16:36:02.085893 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:36:02.086239 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37975 every 8 connection(s)
I20260619 16:36:02.087697 7128 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:36:02.088716 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Bootstrap starting.
I20260619 16:36:02.098918 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:36:02.099236 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: Bootstrap complete.
I20260619 16:36:02.099802 7128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } }
I20260619 16:36:02.099936 7128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e1fed4b2b9d497ebe98f682df9493da, State: Initialized, Role: LEARNER
I20260619 16:36:02.100040 7128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } }
I20260619 16:36:02.100382 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } }
I20260619 16:36:02.100503 7128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:02.100574 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: LEARNER
I20260619 16:36:02.103441 7142 catalog_manager.cc:1259] Loaded cluster ID: b9b51045b86c49e285eaf80ec741b0ab
I20260619 16:36:02.103551 7142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: loading cluster ID for follower catalog manager: success
I20260619 16:36:02.105724 7142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: acquiring CA information for follower catalog manager: success
I20260619 16:36:02.107052 7142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:36:02.154852 7060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:02.155215 7060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:02.155283 7060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:02.159605 7060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:02.159667 7060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:02.159686 7060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:02.159703 7060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:02.159718 7060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:02.165179 7060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:02.166250 7060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:02.167341 7060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:36:02.175990 7060 server_base.cc:1034] running on GCE node
W20260619 16:36:02.175954 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.175954 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.175951 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.176755 7060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:02.177960 7060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:02.179163 7060 hybrid_clock.cc:648] HybridClock initialized: now 1781886962179147 us; error 30 us; skew 500 ppm
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:02.182314 7060 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 16:36:02.183218 7060 webserver.cc:460] Webserver started at http://127.0.98.60:46413/ using document root <none> and password file <none>
I20260619 16:36:02.183799 7060 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.183887 7060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.184111 7060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:02.186167 7060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/instance:
uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
format_stamp: "Formatted at 2026-06-19 16:36:02 on dist-test-slave-0f28"
server_key: "71120a59e35cec3e3457563b997edc66"
server_key_iv: "e9c4cf2dca6b09bade76c10e23bd50c0"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.186712 7060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal/instance:
uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
format_stamp: "Formatted at 2026-06-19 16:36:02 on dist-test-slave-0f28"
server_key: "71120a59e35cec3e3457563b997edc66"
server_key_iv: "e9c4cf2dca6b09bade76c10e23bd50c0"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.190250 7060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:36:02.192467 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.193693 7060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:02.193820 7060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
format_stamp: "Formatted at 2026-06-19 16:36:02 on dist-test-slave-0f28"
server_key: "71120a59e35cec3e3457563b997edc66"
server_key_iv: "e9c4cf2dca6b09bade76c10e23bd50c0"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.193923 7060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.212661 7060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:02.215459 7060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:02.215653 7060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:02.222636 7060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33025
I20260619 16:36:02.222661 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33025 every 8 connection(s)
I20260619 16:36:02.223630 7060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:36:02.225626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 7060
I20260619 16:36:02.226531 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:02.230715 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:02.241158 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.227014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.0.98.60:33025)
0619 16:36:02.227867 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.227884 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.227916 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:36:02.228529 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.228545 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.228854 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.229186 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.229978 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.230938 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.231779 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.232477 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.232665 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.235732 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.235739 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.235751 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.235781 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.237837 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.238369 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.238375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.238381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.238515 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.238896 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.238902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.238903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.239400 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.239603 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.239925 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.240207 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":581,"threads_started":1}
I20260619 16:36:02.242069 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.232379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.0.98.62:36897)
0619 16:36:02.232526 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.232530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.232798 (+ 268us) server_negotiation.cc:407] Connection header received
0619 16:36:02.233031 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.233035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.233096 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.233189 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.234159 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.234709 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.235799 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.236011 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.236139 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.238729 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.238732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.238734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.238756 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.240370 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.240807 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.240810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.240812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.240895 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.241184 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.241187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.241189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.241426 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.241530 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.241869 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.241966 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:36:02.242162 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.232242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:60834)
0619 16:36:02.232669 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.232717 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.232890 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.233274 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.233280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.233798 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.233976 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.233985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.234868 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.234871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.235665 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.235674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.236155 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.236158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.236268 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.236865 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.236885 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.238573 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.240490 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.240496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.240500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.240709 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.241003 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.241005 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.241006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.241083 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.241537 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.241543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.241680 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.241821 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.241973 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":140,"threads_started":1}
I20260619 16:36:02.243217 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:02.251080 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.243622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:41782)
0619 16:36:02.243725 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.243736 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.243791 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.244207 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.244210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.244355 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.244434 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.244439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.245313 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.245339 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.245857 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.245864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.246316 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.246320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.246414 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.246872 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.246883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.247941 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.249696 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.249700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.249702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.249887 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.250169 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.250172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.250173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.250252 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.250607 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.250610 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.250692 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.250806 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.250920 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 16:36:02.251132 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.243737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.0.98.61:37975)
0619 16:36:02.244004 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.244007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.244018 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:36:02.244052 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.244054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.244097 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.244195 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.244547 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.245210 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.245972 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.246209 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.246318 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.248078 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.248087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.248090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.248110 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.249572 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.249984 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.249989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.249991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.250072 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.250338 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.250343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.250344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.250509 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.250618 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.250855 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.250965 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260619 16:36:02.251902 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:36:02.259861 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.252372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44550 (local address 127.0.98.60:33025)
0619 16:36:02.252454 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.252457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.252470 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:02.252549 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.252552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.252585 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.252672 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.252997 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.253884 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.254602 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.254816 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.254918 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.256807 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.256811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.256813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.256861 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.258380 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.258787 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.258793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.258795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.258869 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.259122 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.259128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.259129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.259284 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.259383 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.259601 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.259736 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":27}
I20260619 16:36:02.259861 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.252279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:44550)
0619 16:36:02.252382 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.252397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.252467 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.252681 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.252683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.252832 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.252883 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.252887 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.253988 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.253994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.254474 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.254481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.254915 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.254918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.255027 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.255451 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.255464 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.256681 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.258484 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.258489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.258492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.258696 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.258953 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.258958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.258960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.259037 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.259374 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.259376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.259450 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.259567 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.259687 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":19}
I20260619 16:36:02.262854 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Bootstrap starting.
I20260619 16:36:02.265059 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:02.265818 7210 log.cc:824] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:02.267772 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: No bootstrap required, opened a new log
I20260619 16:36:02.270969 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } }
I20260619 16:36:02.271181 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:02.271263 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1b02b63e5c431d9fd6bb2972bc301a, State: Initialized, Role: FOLLOWER
I20260619 16:36:02.271740 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } }
I20260619 16:36:02.272186 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } } }
I20260619 16:36:02.272358 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:02.272914 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:36:02.277704 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:36:02.277781 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:36:02.281414 7060 master_runner.cc:428] Detected that this master 9c1b02b63e5c431d9fd6bb2972bc301a is joining an existing cluster
I20260619 16:36:02.281472 7060 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:33025
I20260619 16:36:02.292409 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.283654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.0.98.62:36897)
0619 16:36:02.283869 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.283874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.284655 (+ 781us) server_negotiation.cc:407] Connection header received
0619 16:36:02.284817 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.284821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.284868 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.284917 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.285495 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.286031 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287131 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.287447 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287512 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.288931 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.288935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.288938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.288964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.290679 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.291149 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.291153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.291155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.291230 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.291527 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.291531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.291533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.291735 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.291803 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.292136 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.292245 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 16:36:02.293156 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.284022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:60840)
0619 16:36:02.284576 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.284588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.284712 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.285061 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.285066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.285246 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.285380 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.285386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.286153 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.286156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.287013 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.287018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.287651 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.287790 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.288392 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.288404 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.288793 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.290801 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.290804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.290806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.291033 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.291334 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.291337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.291338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.291416 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.291873 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.291875 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.291956 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.292835 (+ 879us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.292967 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":44,"thread_start_us":109,"threads_started":1}
I20260619 16:36:02.294924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6553
I20260619 16:36:02.295217 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.284067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.61:37975)
0619 16:36:02.284170 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.284175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.284263 (+ 88us) server_negotiation.cc:407] Connection header received
0619 16:36:02.284656 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.284660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.284711 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.284782 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.286290 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.287055 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.288628 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.288997 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.289107 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.290275 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.290279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.290281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.290306 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.292281 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.293022 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.293026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.293053 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.293168 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.293720 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.293723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.293725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.294254 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.294374 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.294838 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.295006 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 16:36:02.295756 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.283672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:41784)
0619 16:36:02.284078 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.284198 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.284302 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.284889 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.284892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.285102 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.286163 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.286171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.287206 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.288506 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.288516 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.289105 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.289109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.289196 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.289690 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.289702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.290143 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.292471 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.292474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.292476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.292848 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.293366 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.293369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.293371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.293551 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.294468 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.294471 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.294625 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.295349 (+ 724us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.295553 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":22,"thread_start_us":111,"threads_started":1}
I20260619 16:36:02.313052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37615
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36897
--webserver_port=34647
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:36:02.314354 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.283753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.0.98.60:33025)
0619 16:36:02.283947 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.283952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.284354 (+ 402us) server_negotiation.cc:407] Connection header received
0619 16:36:02.284432 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.284435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.284478 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.284558 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.285098 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.285920 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287121 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.287425 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287504 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.310973 (+ 23469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.310977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.310979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.311000 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.312676 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.313098 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.313101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.313103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.313170 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.313453 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.313456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.313457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.313666 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.313728 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.314051 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.314168 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":136}
I20260619 16:36:02.314874 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.283705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:44552)
0619 16:36:02.284268 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.284280 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.284357 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.284593 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.284597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.284837 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.284992 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.285003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.286036 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.286039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.287009 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.287017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.287651 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.287655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.287825 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.310395 (+ 22570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.310415 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.310851 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.312780 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.312783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.312785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.313009 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.313251 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.313253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.313255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.313349 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.313788 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.313792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.313883 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.314535 (+ 652us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.314657 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":118,"thread_start_us":78,"threads_started":1}
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886960, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 16:36:02.319468 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.309790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.98.60:33025)
0619 16:36:02.310100 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.310113 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.310127 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:02.310240 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.310245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.310304 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.310902 (+ 598us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.311016 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.311862 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.313068 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.313482 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.313893 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.315887 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.315891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.315893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.315917 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.317922 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.318325 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.318330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.318332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.318411 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.318687 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.318691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.318693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.318852 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.318947 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.319210 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.319308 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":121,"threads_started":1}
I20260619 16:36:02.319454 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.308954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:44558)
0619 16:36:02.309061 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.309075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.309151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.310455 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.310458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.310674 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.310786 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.310795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.312014 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.312021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.312913 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.312924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.313607 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.313610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.313710 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.314243 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.314258 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.315745 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.318030 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.318036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.318039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.318232 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.318512 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.318515 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.318517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.318599 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.318952 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.318957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.319051 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.319153 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.319299 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 16:36:02.320317 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:33025: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 33025
}
attrs {
promote: true
}
}
dest_uuid: "47d0f9f98d954ad985e316c65f7a5cac"
cas_config_opid_index: 5
I20260619 16:36:02.320856 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } }
I20260619 16:36:02.321924 7052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.323352 7052 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:33025
I20260619 16:36:02.323532 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.323626 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.324062 7060 master.cc:537] Master@127.0.98.60:33025 shutting down...
I20260619 16:36:02.324524 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:02.326051 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Refusing update from remote peer 47d0f9f98d954ad985e316c65f7a5cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:36:02.326812 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:36:02.328780 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac -> Peer 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60:33025): Couldn't send request to peer 9c1b02b63e5c431d9fd6bb2972bc301a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 16:36:02.329267 7060 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:02.329564 7060 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:02.329672 7060 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: stopping tablet replica
I20260619 16:36:02.332525 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.322045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:41788)
0619 16:36:02.322454 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.322468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.322537 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.328643 (+ 6106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.328647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.328666 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:02.328767 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.328772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.330087 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.330090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.330858 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.330866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.331910 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.331913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.332009 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.332057 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.332131 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.332206 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339}
I20260619 16:36:02.333108 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.322275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.0.98.61:37975)
0619 16:36:02.328260 (+ 5985us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.328265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.328279 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:02.328328 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.328332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.328395 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.328503 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:02.328892 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.329953 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.331000 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.331607 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.331708 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.331849 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.332765 (+ 916us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.332921 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5891}
I20260619 16:36:02.334028 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 LEARNER]: Refusing update from remote peer 47d0f9f98d954ad985e316c65f7a5cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 16:36:02.334529 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 16:36:02.337172 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.341380 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d0f9f98d954ad985e316c65f7a5cac. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.341513 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: LEARNER
I20260619 16:36:02.341614 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: true } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.341703 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: LEARNER
I20260619 16:36:02.342944 7050 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: attempting to promote NON_VOTER 5e1fed4b2b9d497ebe98f682df9493da to VOTER
I20260619 16:36:02.343441 7050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } }
I20260619 16:36:02.344722 7060 master.cc:552] Master@127.0.98.60:33025 shutdown complete.
I20260619 16:36:02.344810 7060 master_runner.cc:305] Clearing existing system tablet
I20260619 16:36:02.345022 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 LEARNER]: Refusing update from remote peer 47d0f9f98d954ad985e316c65f7a5cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 16:36:02.345186 7060 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.345279 7060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.345355 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.345029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:44564)
0619 16:36:02.345178 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.345249 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 16:36:02.345480 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 16:36:02.347034 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.346835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:44574)
0619 16:36:02.346925 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.346970 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:36:02.347169 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac -> Peer 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60:33025): Couldn't send request to peer 9c1b02b63e5c431d9fd6bb2972bc301a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:36:02.347718 7060 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:36:02.347973 7052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.349193 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.349288 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.349529 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.349593 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.350107 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.351761 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.351874 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:02.353497 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.353711 7060 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.353856 7060 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260619 16:36:02.353894 7060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
format_stamp: "Formatted at 2026-06-19 16:36:02 on dist-test-slave-0f28"
server_key: "71120a59e35cec3e3457563b997edc66"
server_key_iv: "e9c4cf2dca6b09bade76c10e23bd50c0"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.355418 7060 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:02.357195 7060 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 16:36:02.357290 7060 log.cc:1196] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 16:36:02.357594 7060 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Deleting consensus metadata
I20260619 16:36:02.357774 7060 master_runner.cc:315] Copying system tablet from 127.0.98.62:36897
I20260619 16:36:02.358836 7060 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:36897
I20260619 16:36:02.366878 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.359383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.0.98.62:36897)
0619 16:36:02.359502 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.359506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.359761 (+ 255us) server_negotiation.cc:407] Connection header received
0619 16:36:02.359895 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.359898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.359940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.360032 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.360430 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.360946 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.361954 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.362247 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.362360 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.363723 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.363727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.363729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.363752 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.365186 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.365657 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.365662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.365664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.365742 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.366016 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.366020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.366022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.366194 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.366297 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.366570 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.366712 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:36:02.366906 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.359343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:60852)
0619 16:36:02.359676 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.359687 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.359805 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.360046 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.360049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.360244 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.360320 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.360325 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.361050 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.361057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.361842 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.361851 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.362470 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.362472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.362540 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.363150 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.363170 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.363596 (+ 426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.365339 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.365344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.365349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.365560 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.365837 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.365840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.365842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.365926 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.366296 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.366299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.366397 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.366517 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.366712 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":77,"threads_started":1}
I20260619 16:36:02.367581 6511 tablet_copy_service.cc:140] P 47d0f9f98d954ad985e316c65f7a5cac: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9c1b02b63e5c431d9fd6bb2972bc301a ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60852)
I20260619 16:36:02.367699 6511 tablet_copy_service.cc:161] P 47d0f9f98d954ad985e316c65f7a5cac: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9c1b02b63e5c431d9fd6bb2972bc301a at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60852: session id = 9c1b02b63e5c431d9fd6bb2972bc301a-00000000000000000000000000000000
I20260619 16:36:02.368531 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 16:36:02.369660 7060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 16:36:02.371353 7060 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: tablet copy: Starting download of 0 data blocks...
I20260619 16:36:02.371479 7060 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: tablet copy: Starting download of 1 WAL segments...
I20260619 16:36:02.372937 7060 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 16:36:02.374696 6511 tablet_copy_service.cc:342] P 47d0f9f98d954ad985e316c65f7a5cac: Request end of tablet copy session 9c1b02b63e5c431d9fd6bb2972bc301a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60852
I20260619 16:36:02.374786 6511 tablet_copy_service.cc:433] P 47d0f9f98d954ad985e316c65f7a5cac: ending tablet copy session 9c1b02b63e5c431d9fd6bb2972bc301a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9c1b02b63e5c431d9fd6bb2972bc301a
I20260619 16:36:02.381698 7060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:02.383368 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.383394 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.383610 7060 server_base.cc:1034] running on GCE node
W20260619 16:36:02.383639 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.383934 7060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:02.384240 7060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:02.385372 7060 hybrid_clock.cc:648] HybridClock initialized: now 1781886962385367 us; error 21 us; skew 500 ppm
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:02.386793 7060 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 16:36:02.387235 7060 webserver.cc:460] Webserver started at http://127.0.98.60:38411/ using document root <none> and password file <none>
I20260619 16:36:02.387415 7060 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.387468 7060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.388473 7060 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:02.389360 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.389595 7060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:02.389683 7060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
format_stamp: "Formatted at 2026-06-19 16:36:02 on dist-test-slave-0f28"
server_key: "71120a59e35cec3e3457563b997edc66"
server_key_iv: "e9c4cf2dca6b09bade76c10e23bd50c0"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.389765 7060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.395776 7060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:02.397670 7060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:02.402505 7060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33025
I20260619 16:36:02.403041 7060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:36:02.403429 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33025 every 8 connection(s)
I20260619 16:36:02.404477 7311 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:36:02.405262 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Bootstrap starting.
I20260619 16:36:02.413477 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:36:02.413757 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Bootstrap complete.
I20260619 16:36:02.414258 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } }
I20260619 16:36:02.414390 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9c1b02b63e5c431d9fd6bb2972bc301a, State: Initialized, Role: LEARNER
I20260619 16:36:02.414510 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } }
I20260619 16:36:02.414861 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.415035 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:02.415099 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: This master's current role is: LEARNER
I20260619 16:36:02.418841 7325 catalog_manager.cc:1259] Loaded cluster ID: b9b51045b86c49e285eaf80ec741b0ab
I20260619 16:36:02.418918 7325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: loading cluster ID for follower catalog manager: success
I20260619 16:36:02.420459 7325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: acquiring CA information for follower catalog manager: success
I20260619 16:36:02.421579 7325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 16:36:02.471899 7237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:02.472244 7237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:02.472308 7237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:02.479058 7237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:02.479139 7237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:02.479224 7237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:02.486847 7237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37615
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34647
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:02.488066 7237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:02.489219 7237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:02.499228 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.499217 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.499341 7237 server_base.cc:1034] running on GCE node
W20260619 16:36:02.499266 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.499886 7237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:02.500530 7237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:02.501693 7237 hybrid_clock.cc:648] HybridClock initialized: now 1781886962501663 us; error 50 us; skew 500 ppm
I20260619 16:36:02.503620 6816 heartbeater.cc:503] Master 127.0.98.62:36897 was elected leader, sending a full tablet report...
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:02.504458 7237 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:36:02.505379 7237 webserver.cc:460] Webserver started at http://127.0.98.1:34647/ using document root <none> and password file <none>
I20260619 16:36:02.505939 7237 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.506014 7237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.509611 7237 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 16:36:02.511623 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.512895 7237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:02.513037 7237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "419a6fb8404345bb876dafd02d76affb"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "21361318235c6c77160f91005c59fcc6"
server_key_iv: "ab151247d047cdcd1c3be88c0f4e6f87"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.513679 7237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.529431 7237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:02.532341 7237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:02.532538 7237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:02.533097 7237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:02.533964 7237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:02.534037 7237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.534111 7237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:02.534154 7237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.543047 7237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37615
I20260619 16:36:02.543112 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37615 every 8 connection(s)
I20260619 16:36:02.544049 7237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:02.552601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 7237
I20260619 16:36:02.552800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6687
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:02.559444 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.546078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33997 (local address 127.0.98.60:33025)
0619 16:36:02.546335 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.546341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.547529 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 16:36:02.548099 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.548102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.548148 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.548210 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.549771 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.550467 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.551588 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.551835 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.551949 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.555274 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.555277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.555279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.555303 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.557129 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.557799 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.557803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.557806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.557894 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.558238 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.558242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.558243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.558421 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.558532 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.559115 (+ 583us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.559241 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260619 16:36:02.560349 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.546305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57545 (local address 127.0.98.62:36897)
0619 16:36:02.546445 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.546450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.546971 (+ 521us) server_negotiation.cc:407] Connection header received
0619 16:36:02.548075 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.548078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.548136 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.548219 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.549795 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.550602 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.551996 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.552339 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.552414 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.556703 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.556710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.556716 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.556746 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.558637 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.559062 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.559068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.559073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.559162 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.559458 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.559464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.559469 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.559703 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.559805 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.560097 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.560208 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:36:02.560923 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.546300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:57545)
0619 16:36:02.546781 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.546830 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.547866 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.548387 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.548394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.548943 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.549642 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.549651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.550735 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.550738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.551856 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.551866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.554124 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.554130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.554256 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.554824 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.554844 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.556551 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.558739 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.558742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.558744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.558965 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.559267 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.559270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.559271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.559354 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.559812 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.559815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.559905 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.560488 (+ 583us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.560617 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":219,"spinlock_wait_cycles":45824,"thread_start_us":542,"threads_started":1}
I20260619 16:36:02.560923 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.546372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.98.1:33997)
0619 16:36:02.547399 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.547411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.547865 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.548385 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.548392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.548943 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.549630 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.549645 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.550612 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.550615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.551462 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.551471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.551993 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.551997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.552204 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.553295 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.553349 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.555096 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.557257 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.557265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.557280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.557680 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.558006 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.558008 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.558009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.558131 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.558529 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.558535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.558813 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.559586 (+ 773us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.559888 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":106,"spinlock_wait_cycles":2432,"thread_start_us":146,"threads_started":1}
I20260619 16:36:02.560981 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39291
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36897
--webserver_port=38517
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:36:02.562418 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33025
I20260619 16:36:02.562418 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36897
I20260619 16:36:02.562692 7454 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.562696 7455 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.563259 7455 heartbeater.cc:511] Master 127.0.98.62:36897 requested a full tablet report, sending...
I20260619 16:36:02.563259 7454 heartbeater.cc:511] Master 127.0.98.60:33025 requested a full tablet report, sending...
I20260619 16:36:02.564525 6491 ts_manager.cc:194] Re-registered known tserver with Master: 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615)
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:02.564879 7275 ts_manager.cc:194] Registered new tserver with Master: 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615)
I20260619 16:36:02.565301 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57545
I20260619 16:36:02.568928 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.545930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50583 (local address 127.0.98.61:37975)
0619 16:36:02.546109 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.546114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.547581 (+ 1467us) server_negotiation.cc:407] Connection header received
0619 16:36:02.548177 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.548180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.548241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.548311 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.549793 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.550591 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.551995 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.552339 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.552414 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.565389 (+ 12975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.565393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.565396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.565423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.567155 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.567706 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.567710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.567711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.567792 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.568126 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.568128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.568129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.568304 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.568397 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.568702 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.568800 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:36:02.569178 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.546300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.98.1:50583)
0619 16:36:02.547496 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.547510 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.547867 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.548426 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.548429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.548943 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.549642 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.549651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.550735 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.550738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.551858 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.551866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.552574 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.552578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.552686 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.563178 (+ 10492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.563191 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.565182 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.567261 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.567267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.567272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.567561 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.567912 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.567917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.567921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.568025 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.568398 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.568403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.568528 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.568653 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.568838 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":1149,"spinlock_wait_cycles":21760,"thread_start_us":98,"threads_started":1}
I20260619 16:36:02.569928 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37975
I20260619 16:36:02.570034 7456 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.570238 7456 heartbeater.cc:511] Master 127.0.98.61:37975 requested a full tablet report, sending...
I20260619 16:36:02.571251 7083 ts_manager.cc:194] Registered new tserver with Master: 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615)
W20260619 16:36:02.676013 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:02.676316 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:02.676400 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:02.680681 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:02.680766 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:02.680862 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:02.686282 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38517
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:02.687350 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:02.688395 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:02.698539 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.698504 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.698508 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.698570 7462 server_base.cc:1034] running on GCE node
I20260619 16:36:02.699122 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:02.699779 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:02.700935 7462 hybrid_clock.cc:648] HybridClock initialized: now 1781886962700915 us; error 30 us; skew 500 ppm
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:02.703862 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:36:02.704720 7462 webserver.cc:460] Webserver started at http://127.0.98.2:38517/ using document root <none> and password file <none>
I20260619 16:36:02.705261 7462 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.705394 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.708956 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:02.710843 6950 heartbeater.cc:503] Master 127.0.98.62:36897 was elected leader, sending a full tablet report...
I20260619 16:36:02.710996 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.712431 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:36:02.712551 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "bdd289fed93242188e10f130ae14e046"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "d5e1b5efad26e7baa0c30c0f0a24fbca"
server_key_iv: "53c399c1beec7cf6de0ac4b51b727fc5"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.713114 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.726764 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:02.729784 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:02.729966 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:02.730501 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:02.731313 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:02.731384 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.731451 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:02.731503 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.739926 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39291
I20260619 16:36:02.739952 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39291 every 8 connection(s)
I20260619 16:36:02.740880 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:02.748569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 7462
I20260619 16:36:02.748759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6821
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:02.758237 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.743213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53575 (local address 127.0.98.60:33025)
0619 16:36:02.743345 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.743351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.744572 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 16:36:02.745025 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.745031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.745089 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.745171 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.746551 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.747361 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.749689 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.750018 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.750086 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.752945 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.752949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.752952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.752982 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.755248 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756106 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.756110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.756112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.756197 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756556 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.756560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.756562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.756786 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.756845 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.757925 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.758049 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:36:02.758457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45481
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36897
--webserver_port=46163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:36:02.758517 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.742812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50473 (local address 127.0.98.61:37975)
0619 16:36:02.743025 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.743029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.744335 (+ 1306us) server_negotiation.cc:407] Connection header received
0619 16:36:02.745286 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.745289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.745378 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.745460 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.746539 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.747066 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.748825 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.749233 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.749375 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.753370 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.753377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.753383 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.753429 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.755428 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756104 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.756111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.756115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.756200 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756568 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.756574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.756579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.756826 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.756921 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.758194 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.758359 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260619 16:36:02.759162 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.742683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34929 (local address 127.0.98.62:36897)
0619 16:36:02.742939 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.742944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.743883 (+ 939us) server_negotiation.cc:407] Connection header received
0619 16:36:02.744930 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.744934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.744996 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.745078 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.746560 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.747361 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.749480 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.749821 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.749909 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.753273 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.753277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.753281 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.753311 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.755395 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756047 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.756051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.756054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.756142 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756556 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.756560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.756561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.757026 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.757114 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.758901 (+ 1787us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.758991 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260619 16:36:02.760329 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.743170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.98.2:50473)
0619 16:36:02.744246 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.744260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.744740 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.745490 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.745493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.745845 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.746393 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.746407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.747215 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.747219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.748659 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.748670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.749572 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.749578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.749830 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.750856 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.750877 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.753190 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.755566 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.755572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.755577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.755993 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.756311 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.756322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.756452 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.757026 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.757035 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.757404 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.759062 (+ 1658us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.759289 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":157,"spinlock_wait_cycles":65280,"thread_start_us":113,"threads_started":1}
I20260619 16:36:02.760840 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.743225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.98.2:53575)
0619 16:36:02.744479 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.744489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.744741 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.745276 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.745285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.745854 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.746404 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.746413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.747473 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.747476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.749581 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.749587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.750272 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.750278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.750365 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.750818 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.750832 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.752721 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.755436 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.755445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.755461 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.756004 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.756301 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.756310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.756457 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.757632 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.757635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.757735 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.757874 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.758190 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":1254,"spinlock_wait_cycles":34944,"thread_start_us":110,"threads_started":1}
I20260619 16:36:02.761541 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37975
I20260619 16:36:02.761574 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33025
I20260619 16:36:02.761794 7591 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.761794 7593 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.761941 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.743147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:34929)
0619 16:36:02.743738 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.743788 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.744723 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.745277 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.745285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.745854 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.746407 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.746413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.748483 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.748486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.749372 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.749382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.750027 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.750030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.750124 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.750557 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.750583 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.753106 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.755499 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.755504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.755509 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.755922 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.756261 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.756267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.756271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.756442 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.757200 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.757203 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.757407 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.759462 (+ 2055us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.759591 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":129,"spinlock_wait_cycles":39296,"thread_start_us":143,"threads_started":1}
I20260619 16:36:02.762414 7593 heartbeater.cc:511] Master 127.0.98.61:37975 requested a full tablet report, sending...
I20260619 16:36:02.762414 7591 heartbeater.cc:511] Master 127.0.98.60:33025 requested a full tablet report, sending...
I20260619 16:36:02.762598 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.757535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:44584)
0619 16:36:02.757655 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.757669 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.757736 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.758802 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.758808 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.758833 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:02.758947 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.758956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.759930 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.759933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.761059 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.761072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.762204 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.762207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.762293 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.762319 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.762414 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.762481 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:36:02.762792 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.757888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.0.98.60:33025)
0619 16:36:02.758405 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.758409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.758421 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:36:02.758457 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.758460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.758515 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.758607 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:02.759071 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.759804 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.761205 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.761858 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.761974 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.762139 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.762427 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.762543 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":64,"threads_started":1}
I20260619 16:36:02.763221 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36897
I20260619 16:36:02.763299 7592 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.763461 7592 heartbeater.cc:511] Master 127.0.98.62:36897 requested a full tablet report, sending...
I20260619 16:36:02.763554 7275 ts_manager.cc:194] Registered new tserver with Master: bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:02.763717 7083 ts_manager.cc:194] Registered new tserver with Master: bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:02.764007 6491 ts_manager.cc:194] Re-registered known tserver with Master: bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:02.763949 7285 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 16:36:02.764293 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d0f9f98d954ad985e316c65f7a5cac. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.764416 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: This master's current role is: LEARNER
I20260619 16:36:02.764683 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34929
W20260619 16:36:02.877759 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:02.878021 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:02.878068 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:02.882093 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:02.882144 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:02.882201 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:02.887461 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46163
--tserver_master_addrs=127.0.98.62:36897,127.0.98.61:37975,127.0.98.60:33025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:02.888456 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:02.889542 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:02.899134 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:02.899150 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.899561 7599 server_base.cc:1034] running on GCE node
W20260619 16:36:02.899151 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:02.899945 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:02.900544 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:02.901777 7599 hybrid_clock.cc:648] HybridClock initialized: now 1781886962901755 us; error 32 us; skew 500 ppm
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:02.904726 7599 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:36:02.905731 7599 webserver.cc:460] Webserver started at http://127.0.98.3:46163/ using document root <none> and password file <none>
I20260619 16:36:02.906314 7599 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:02.906440 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:02.910625 7599 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:36:02.912753 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.914086 7599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:02.914227 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "1a982f53b199477faf9d8df3904c89a2"
format_stamp: "Formatted at 2026-06-19 16:36:01 on dist-test-slave-0f28"
server_key: "658efb6ad19116ca20147b8e04ab5118"
server_key_iv: "e249dc5334b4f3ac6ce1f038fb421893"
server_key_version: "encryptionkey@0"
I20260619 16:36:02.914857 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:02.928544 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:02.931741 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:02.931948 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:02.932520 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:02.933475 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:02.933543 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.933605 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:02.933656 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:02.942550 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45481
I20260619 16:36:02.942580 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45481 every 8 connection(s)
I20260619 16:36:02.943506 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:02.948025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 7599
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 16:36:02 dist-test-slave-0f28 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 16:36:02.963564 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.945465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41315 (local address 127.0.98.61:37975)
0619 16:36:02.945746 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.945751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.946920 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 16:36:02.947833 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.947837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.947890 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.947953 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.949939 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.950745 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.954006 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.954523 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.954618 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.959642 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.959649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.959655 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.959685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.961605 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.962086 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.962089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.962091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.962179 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.962502 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.962509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.962511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.962754 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.962849 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.963223 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.963352 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":197}
I20260619 16:36:02.964424 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.946033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35777 (local address 127.0.98.60:33025)
0619 16:36:02.946155 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.946161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.946794 (+ 633us) server_negotiation.cc:407] Connection header received
0619 16:36:02.947833 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.947837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.947892 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.947948 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.949928 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.950745 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.952680 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.953006 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.953109 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.957403 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.957408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.957410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.957439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.960675 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.961404 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.961409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.961411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.961511 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.961977 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.961981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.961983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.962239 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.962343 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.964132 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.964254 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:36:02.964435 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.945615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34241 (local address 127.0.98.62:36897)
0619 16:36:02.945842 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.945846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.946563 (+ 717us) server_negotiation.cc:407] Connection header received
0619 16:36:02.947879 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.947884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.947950 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.948013 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.949946 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.950806 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.952719 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.953155 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.953250 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.957403 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.957407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.957410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.957434 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.959319 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.963046 (+ 3727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.963056 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.963058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.963147 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.963490 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.963494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.963496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.963688 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.963785 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.964146 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.964254 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260619 16:36:02.964855 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.945815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.98.3:41315)
0619 16:36:02.946830 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.946842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.947598 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.948357 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.948366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.948986 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.949778 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.949783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.950909 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.950912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.953879 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.953889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.955511 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.955515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.955602 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.957790 (+ 2188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.957809 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.959479 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.961737 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.961742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.961745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.961983 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.962297 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.962303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.962308 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.962399 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.962896 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.962901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.963018 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.963803 (+ 785us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.963958 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":687,"spinlock_wait_cycles":5632,"thread_start_us":105,"threads_started":1}
I20260619 16:36:02.965598 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.945815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.98.3:35777)
0619 16:36:02.946685 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.946698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.947598 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.948357 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.948366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.948989 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.949763 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.949782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.950909 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.950912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.952553 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.952566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.953290 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.953296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.953562 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.954336 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.954351 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.957164 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.960807 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.960815 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.960834 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.961243 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.961644 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.961647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.961649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.961844 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.962405 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.962413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.962767 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.962997 (+ 230us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.963370 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":258,"spinlock_wait_cycles":33792,"thread_start_us":115,"threads_started":1}
I20260619 16:36:02.966007 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.945815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:34241)
0619 16:36:02.946403 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.946471 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.947634 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.948362 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.948366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.948970 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:02.949776 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.949789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.950962 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.950967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.952582 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.952593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.953494 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.953498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.953625 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.954279 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:02.954300 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:02.957179 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:02.962669 (+ 5490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.962672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.962674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.962924 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.963266 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:02.963269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:02.963271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:02.963362 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:02.963852 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:02.963856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:02.963953 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.964640 (+ 687us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.964777 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":352,"spinlock_wait_cycles":7424,"thread_start_us":153,"threads_started":1}
I20260619 16:36:02.966403 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.950997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.0.98.61:37975)
0619 16:36:02.952258 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.952263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.952278 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:36:02.952321 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.952331 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.952387 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.953626 (+ 1239us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.953747 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.954546 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.956776 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.957180 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.957332 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.961962 (+ 4630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.961966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.961968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.961994 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.964101 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.964573 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.964576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.964578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.964658 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.964939 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.964942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.964944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.965129 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.965425 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.966145 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.966262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1183,"thread_start_us":106,"threads_started":1}
I20260619 16:36:02.966598 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.61:37975
I20260619 16:36:02.966624 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36897
I20260619 16:36:02.966826 7730 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.966826 7729 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.967419 7730 heartbeater.cc:511] Master 127.0.98.61:37975 requested a full tablet report, sending...
I20260619 16:36:02.967417 7729 heartbeater.cc:511] Master 127.0.98.62:36897 requested a full tablet report, sending...
I20260619 16:36:02.967890 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.60:33025
I20260619 16:36:02.967980 7728 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:02.968149 7728 heartbeater.cc:511] Master 127.0.98.60:33025 requested a full tablet report, sending...
I20260619 16:36:02.968444 6491 ts_manager.cc:194] Re-registered known tserver with Master: 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3:45481)
I20260619 16:36:02.968793 7083 ts_manager.cc:194] Registered new tserver with Master: 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3:45481)
I20260619 16:36:02.969167 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34241
I20260619 16:36:02.969237 7275 ts_manager.cc:194] Registered new tserver with Master: 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3:45481)
I20260619 16:36:02.970256 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.951843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.0.98.62:36897)
0619 16:36:02.955350 (+ 3507us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.955354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.955746 (+ 392us) server_negotiation.cc:407] Connection header received
0619 16:36:02.955785 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.955788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.955840 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.955924 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.956876 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.957684 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.958940 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.959277 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.959360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.962063 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.962070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.962073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.962097 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.968078 (+ 5981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.968576 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.968582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.968584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.968676 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.968974 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.968977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.968979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.969151 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.969307 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.969866 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.970102 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3424,"thread_start_us":64,"threads_started":1}
I20260619 16:36:02.973837 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.951239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.0.98.60:33025)
0619 16:36:02.951391 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.951398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.952584 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 16:36:02.952752 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.952755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.952795 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.952846 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:02.953771 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.954999 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.959142 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.959469 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.959537 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.969949 (+ 10412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:02.969953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:02.969956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:02.969982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:02.971828 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.972377 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.972380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.972382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.972464 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:02.972847 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:02.972849 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:02.972850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:02.973111 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:02.973220 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.973566 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.973684 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:36:02.973811 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:02.976665 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:02.979595 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: true } } }
I20260619 16:36:02.979761 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.983109 7746 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: attempting to promote NON_VOTER 9c1b02b63e5c431d9fd6bb2972bc301a to VOTER
I20260619 16:36:02.983522 7746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:02.984712 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Refusing update from remote peer 47d0f9f98d954ad985e316c65f7a5cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:36:02.984712 7285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 LEARNER]: Refusing update from remote peer 47d0f9f98d954ad985e316c65f7a5cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 16:36:02.985093 7745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 16:36:02.985611 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 16:36:02.986640 7746 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.987388 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.988180 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.988274 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.987942 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.988387 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.988482 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.988437 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.988569 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:02.989156 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.989282 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:02.990298 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47d0f9f98d954ad985e316c65f7a5cac" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:02.990437 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:02.997867 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.992958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.0.1:47008)
0619 16:36:02.993102 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.993117 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.993196 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.994080 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.994086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.994107 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:02.994215 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.994225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.995724 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.995729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.996591 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.996600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.997431 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.997434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.997567 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.997591 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:36:02.997672 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:36:02.997744 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":24}
I20260619 16:36:02.998169 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.993139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.0.98.1:37615)
0619 16:36:02.993546 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.993562 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.993590 (+ 28us) server_negotiation.cc:407] Connection header received
0619 16:36:02.993668 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.993673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.993900 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.994381 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:02.994540 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.995592 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.996718 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.997263 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.997432 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:02.997585 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:02.997701 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:36:02.998015 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":110,"threads_started":1}
I20260619 16:36:03.001685 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 79ad6e97b5f749dca6998663e5803b96 (DEFAULT_TABLE table=test-table [id=13c497ede3df4a969408939225f5c9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:03.002803 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ad6e97b5f749dca6998663e5803b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:03.007714 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.993722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39291 (local address 127.0.0.1:54908)
0619 16:36:02.995165 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.995180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.995271 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.996006 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.996011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.996036 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:02.996134 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.996142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.997306 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.997309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:02.999549 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.999560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.000712 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.000715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.007351 (+ 6636us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.007376 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:36:03.007482 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:36:03.007582 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":6514,"thread_start_us":91,"threads_started":1}
I20260619 16:36:03.007764 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.994429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.0.98.2:39291)
0619 16:36:02.995503 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.995513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.995531 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:36:02.995592 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.995596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.995818 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.996300 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:02.996412 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.997169 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:02.999708 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.000418 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.000510 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.000628 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.007521 (+ 6893us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.007627 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":115,"threads_started":1}
I20260619 16:36:03.010208 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.994034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45481 (local address 127.0.0.1:55198)
0619 16:36:02.994299 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:02.994314 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:02.994384 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:02.998191 (+ 3807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:02.998194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:02.998212 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:02.998309 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:02.998316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.000039 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.000042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.008522 (+ 8480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:03.008532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.009840 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.009844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.009927 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.009950 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:36:03.010020 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:36:03.010088 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260619 16:36:03.010713 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:02.994621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.0.98.3:45481)
0619 16:36:02.997546 (+ 2925us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:02.997558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:02.997581 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:36:02.997659 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:02.997665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:02.997864 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:02.998041 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:02.998447 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:02.999660 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.008670 (+ 9010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.009546 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.009649 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.009779 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.010475 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.010582 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2809,"thread_start_us":95,"threads_started":1}
I20260619 16:36:03.011065 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 79ad6e97b5f749dca6998663e5803b96 (DEFAULT_TABLE table=test-table [id=13c497ede3df4a969408939225f5c9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:03.012158 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ad6e97b5f749dca6998663e5803b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:03.012076 7760 tablet_bootstrap.cc:492] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Bootstrap starting.
I20260619 16:36:03.013679 7662 tablet_service.cc:1467] Processing CreateTablet for tablet 79ad6e97b5f749dca6998663e5803b96 (DEFAULT_TABLE table=test-table [id=13c497ede3df4a969408939225f5c9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:03.014645 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ad6e97b5f749dca6998663e5803b96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:03.015174 7760 tablet_bootstrap.cc:654] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:03.016146 7760 log.cc:824] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:03.018267 7760 tablet_bootstrap.cc:492] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: No bootstrap required, opened a new log
I20260619 16:36:03.018476 7760 ts_tablet_manager.cc:1397] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:36:03.020815 7762 tablet_bootstrap.cc:492] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Bootstrap starting.
I20260619 16:36:03.023070 7762 tablet_bootstrap.cc:654] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:03.023241 7760 raft_consensus.cc:348] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.023561 7760 raft_consensus.cc:374] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:03.023617 7760 raft_consensus.cc:729] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419a6fb8404345bb876dafd02d76affb, State: Initialized, Role: FOLLOWER
I20260619 16:36:03.023890 7762 log.cc:824] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:03.024107 7760 consensus_queue.cc:260] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.024567 7763 tablet_bootstrap.cc:492] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Bootstrap starting.
I20260619 16:36:03.025143 7760 ts_tablet_manager.cc:1428] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 16:36:03.025496 7455 heartbeater.cc:503] Master 127.0.98.62:36897 was elected leader, sending a full tablet report...
I20260619 16:36:03.026036 7762 tablet_bootstrap.cc:492] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: No bootstrap required, opened a new log
I20260619 16:36:03.026244 7762 ts_tablet_manager.cc:1397] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:36:03.030805 7763 tablet_bootstrap.cc:654] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:03.031070 7762 raft_consensus.cc:348] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.031293 7762 raft_consensus.cc:374] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:03.031358 7762 raft_consensus.cc:729] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd289fed93242188e10f130ae14e046, State: Initialized, Role: FOLLOWER
I20260619 16:36:03.031392 7763 log.cc:824] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:03.031771 7762 consensus_queue.cc:260] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.032413 7592 heartbeater.cc:503] Master 127.0.98.62:36897 was elected leader, sending a full tablet report...
I20260619 16:36:03.032702 7762 ts_tablet_manager.cc:1428] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:36:03.033527 7763 tablet_bootstrap.cc:492] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: No bootstrap required, opened a new log
I20260619 16:36:03.033703 7763 ts_tablet_manager.cc:1397] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 16:36:03.037578 7763 raft_consensus.cc:348] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.037808 7763 raft_consensus.cc:374] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:03.037864 7763 raft_consensus.cc:729] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a982f53b199477faf9d8df3904c89a2, State: Initialized, Role: FOLLOWER
I20260619 16:36:03.038244 7763 consensus_queue.cc:260] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.038885 7729 heartbeater.cc:503] Master 127.0.98.62:36897 was elected leader, sending a full tablet report...
I20260619 16:36:03.039124 7763 ts_tablet_manager.cc:1428] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 16:36:03.046016 7457 tablet.cc:2369] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:36:03.193884 7768 raft_consensus.cc:482] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:03.194069 7768 raft_consensus.cc:504] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
W20260619 16:36:03.194844 7731 tablet.cc:2369] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:36:03.195297 7768 leader_election.cc:288] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615), bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:03.198920 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.195494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.98.3:58385)
0619 16:36:03.195617 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:03.195634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:03.195727 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:03.196095 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:03.196097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:03.196112 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:03.196203 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:03.196208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.196937 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.196940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.197780 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:03.197788 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.198418 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.198422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.198516 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.198590 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 16:36:03.198704 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:36:03.198769 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":35}
I20260619 16:36:03.198911 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.195614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58385 (local address 127.0.98.1:37615)
0619 16:36:03.195753 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.195758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.195774 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:36:03.195831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.195834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.195885 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.195983 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:03.196317 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.196819 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.197904 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.198304 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.198420 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.198482 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.198717 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.198783 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:36:03.198969 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.195883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35249 (local address 127.0.98.2:39291)
0619 16:36:03.196035 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.196041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.196056 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:36:03.196108 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.196110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.196148 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.196231 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:03.196485 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.196973 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.198016 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.198400 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.198517 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.198579 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.198771 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.198842 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:36:03.198969 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.195771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39291 (local address 127.0.98.3:35249)
0619 16:36:03.195878 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:03.195891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:03.195979 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:03.196240 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:03.196243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:03.196259 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:03.196368 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:03.196373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.197084 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.197090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.197887 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:03.197895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.198518 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.198521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:03.198652 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.198667 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 16:36:03.198754 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:36:03.198823 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":5}
I20260619 16:36:03.199541 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ad6e97b5f749dca6998663e5803b96" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419a6fb8404345bb876dafd02d76affb" is_pre_election: true
I20260619 16:36:03.199548 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ad6e97b5f749dca6998663e5803b96" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd289fed93242188e10f130ae14e046" is_pre_election: true
I20260619 16:36:03.199813 7408 raft_consensus.cc:2454] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 0.
I20260619 16:36:03.199832 7545 raft_consensus.cc:2454] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 0.
I20260619 16:36:03.200242 7617 leader_election.cc:302] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a982f53b199477faf9d8df3904c89a2, 419a6fb8404345bb876dafd02d76affb; no voters:
I20260619 16:36:03.200529 7768 raft_consensus.cc:2781] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:03.200650 7768 raft_consensus.cc:482] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:03.200716 7768 raft_consensus.cc:3037] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:03.201924 7768 raft_consensus.cc:504] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.202219 7768 leader_election.cc:288] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 election: Requested vote from peers 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615), bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:03.202610 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ad6e97b5f749dca6998663e5803b96" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd289fed93242188e10f130ae14e046"
I20260619 16:36:03.202569 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ad6e97b5f749dca6998663e5803b96" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419a6fb8404345bb876dafd02d76affb"
I20260619 16:36:03.202735 7545 raft_consensus.cc:3037] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:03.202756 7408 raft_consensus.cc:3037] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:03.203725 7408 raft_consensus.cc:2454] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 1.
I20260619 16:36:03.203735 7545 raft_consensus.cc:2454] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 1.
I20260619 16:36:03.203994 7617 leader_election.cc:302] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a982f53b199477faf9d8df3904c89a2, bdd289fed93242188e10f130ae14e046; no voters:
I20260619 16:36:03.204152 7768 raft_consensus.cc:2781] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:03.204360 7768 raft_consensus.cc:686] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a982f53b199477faf9d8df3904c89a2, State: Running, Role: LEADER
I20260619 16:36:03.204840 7768 consensus_queue.cc:237] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } }
I20260619 16:36:03.208220 6490 catalog_manager.cc:5495] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a982f53b199477faf9d8df3904c89a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } health_report { overall_health: HEALTHY } } }
I20260619 16:36:03.232648 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.229143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.0.98.3:45481)
0619 16:36:03.229293 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.229297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.229312 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:36:03.229506 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.229514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.229572 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.229678 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:03.229904 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.230397 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.231262 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.231477 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.231582 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.231872 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:03.232259 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:03.232378 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.232453 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.232509 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:36:03.238338 7408 raft_consensus.cc:1261] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Refusing update from remote peer 1a982f53b199477faf9d8df3904c89a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:03.238337 7545 raft_consensus.cc:1261] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Refusing update from remote peer 1a982f53b199477faf9d8df3904c89a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:03.239086 7768 consensus_queue.cc:1035] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:03.239318 7769 consensus_queue.cc:1035] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:36:03.242872 7594 tablet.cc:2369] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:36:03.258160 6490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:03.258342 6490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:03.262689 6490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47d0f9f98d954ad985e316c65f7a5cac: Sending DeleteTablet for 3 replicas of tablet 79ad6e97b5f749dca6998663e5803b96
I20260619 16:36:03.263682 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ad6e97b5f749dca6998663e5803b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47008
I20260619 16:36:03.264166 7782 tablet_replica.cc:330] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: stopping tablet replica
I20260619 16:36:03.264335 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ad6e97b5f749dca6998663e5803b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55198
I20260619 16:36:03.264657 7782 raft_consensus.cc:2229] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:03.264904 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ad6e97b5f749dca6998663e5803b96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54908
I20260619 16:36:03.265089 7782 raft_consensus.cc:2258] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:03.265858 7785 tablet_replica.cc:330] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: stopping tablet replica
I20260619 16:36:03.266119 7785 raft_consensus.cc:2229] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:03.266387 7785 raft_consensus.cc:2258] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:03.266832 7782 ts_tablet_manager.cc:1905] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:03.266924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6459
I20260619 16:36:03.268272 7785 ts_tablet_manager.cc:1905] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:03.269800 7782 ts_tablet_manager.cc:1918] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:03.269974 7782 log.cc:1196] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/79ad6e97b5f749dca6998663e5803b96
I20260619 16:36:03.270363 7782 ts_tablet_manager.cc:1939] T 79ad6e97b5f749dca6998663e5803b96 P 419a6fb8404345bb876dafd02d76affb: Deleting consensus metadata
I20260619 16:36:03.270939 7785 ts_tablet_manager.cc:1918] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:03.271034 7785 log.cc:1196] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/79ad6e97b5f749dca6998663e5803b96
I20260619 16:36:03.271332 7785 ts_tablet_manager.cc:1939] T 79ad6e97b5f749dca6998663e5803b96 P bdd289fed93242188e10f130ae14e046: Deleting consensus metadata
I20260619 16:36:03.274331 7783 tablet_replica.cc:330] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: stopping tablet replica
I20260619 16:36:03.274621 7783 raft_consensus.cc:2229] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:03.274916 7783 raft_consensus.cc:2258] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:36:03.275794 7481 connection.cc:535] server connection from 127.0.0.1:54908 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 16:36:03.276181 7343 connection.cc:535] server connection from 127.0.0.1:47008 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:36:03.276434 7783 ts_tablet_manager.cc:1905] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:03.279357 7783 ts_tablet_manager.cc:1918] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:03.279470 7783 log.cc:1196] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/79ad6e97b5f749dca6998663e5803b96
I20260619 16:36:03.279805 7783 ts_tablet_manager.cc:1939] T 79ad6e97b5f749dca6998663e5803b96 P 1a982f53b199477faf9d8df3904c89a2: Deleting consensus metadata
W20260619 16:36:03.280848 7618 connection.cc:441] server connection from 127.0.0.1:55198 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55198 (request call id 1) could send its response
I20260619 16:36:03.282655 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.265627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.98.61:37975)
0619 16:36:03.265732 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.265736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.265749 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:03.265786 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.265789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.265830 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.265919 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:03.274175 (+ 8256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.274920 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.276225 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.276561 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.276637 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.278718 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:03.278725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:03.278728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:03.278753 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:03.280663 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.281205 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.281209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.281211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.281297 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.281645 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.281648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.281650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.281872 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:03.281999 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.282359 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.282501 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:36:03.290923 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.280114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.0.98.60:33025)
0619 16:36:03.280218 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.280222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.281109 (+ 887us) server_negotiation.cc:407] Connection header received
0619 16:36:03.281251 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.281257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.281308 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.281412 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:03.282398 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.283158 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.284404 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.284713 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.284782 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.286775 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:03.286778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:03.286781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:03.286805 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:03.288918 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.289404 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.289407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.289409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.289547 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.289883 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.289889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.289893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.290140 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:03.290251 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.290644 (+ 393us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.290747 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:36:03.291786 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.281568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41830 (local address 127.0.98.61:37975)
0619 16:36:03.281690 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.281696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.281748 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:36:03.281914 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.281917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.281959 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.282007 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:03.282826 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.283592 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.285543 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.285870 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.285955 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.287515 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:03.287519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:03.287521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:03.287545 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:03.289949 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.290432 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.290436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.290438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.290531 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.290855 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.290858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.290860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.291058 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:03.291173 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.291535 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.291637 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:36:03.292879 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:03.265263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.0.98.60:33025)
0619 16:36:03.265398 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:03.265403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:03.283349 (+ 17946us) server_negotiation.cc:407] Connection header received
0619 16:36:03.283507 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:03.283510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:03.283551 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:03.283598 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:03.284421 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.285114 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.286757 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:03.287122 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:03.287215 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:03.288899 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:03.288902 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:03.288905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:03.288925 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:03.290675 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.291400 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.291403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.291407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.291499 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:03.291798 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:03.291801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:03.291803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:03.292003 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:03.292089 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:03.292609 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 16:36:03.292723 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:36:04.251606 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.251153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:44379)
0619 16:36:04.251404 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.251515 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260619 16:36:04.251747 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.251298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:40809)
0619 16:36:04.251540 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.251623 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
W20260619 16:36:04.251910 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36897 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
W20260619 16:36:04.252074 7729 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36897 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.255335 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.254843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:57841)
0619 16:36:04.255153 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.255251 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
W20260619 16:36:04.255604 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36897 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.763278 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47d0f9f98d954ad985e316c65f7a5cac)
I20260619 16:36:04.763448 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47d0f9f98d954ad985e316c65f7a5cac)
I20260619 16:36:04.763540 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:04.763645 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:04.764550 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897), 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61:37975)
I20260619 16:36:04.764900 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897), 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60:33025)
I20260619 16:36:04.765427 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.764983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44664)
0619 16:36:04.765214 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.765293 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
I20260619 16:36:04.765447 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.764688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44656)
0619 16:36:04.765251 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.765350 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":69,"threads_started":1}
W20260619 16:36:04.765588 7079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
W20260619 16:36:04.765601 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.766506 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.766311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44680)
0619 16:36:04.766397 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.766445 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 16:36:04.766639 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.767164 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.766924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44690)
0619 16:36:04.767030 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.767086 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:36:04.767288 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.769550 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.765012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37975 (local address 127.0.0.1:51794)
0619 16:36:04.765294 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.765307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:04.765421 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:04.766037 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:04.766042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:04.766058 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:04.766149 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:04.766156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.767126 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.767129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:04.768380 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:04.768390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.769090 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.769093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:04.769197 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:04.769214 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:36:04.769305 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 16:36:04.769389 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":49,"threads_started":1}
I20260619 16:36:04.769590 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.764850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.0.98.60:33025)
0619 16:36:04.765077 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:04.765082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:04.765277 (+ 195us) server_negotiation.cc:407] Connection header received
0619 16:36:04.765741 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:04.765745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:04.765799 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:04.765859 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:04.766255 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.767000 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.768617 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.769023 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.769139 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:04.769194 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:04.769371 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:36:04.769442 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260619 16:36:04.769591 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.764723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33025 (local address 127.0.0.1:36052)
0619 16:36:04.765196 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.765210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:04.765603 (+ 393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:04.766013 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:04.766015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:04.766030 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:04.766098 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:04.766102 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.767087 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.767093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:04.768497 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:04.768509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.769140 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.769143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:04.769232 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:04.769257 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:36:04.769366 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:36:04.769447 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260619 16:36:04.769920 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.764806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.98.61:37975)
0619 16:36:04.765099 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:04.765104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:04.765638 (+ 534us) server_negotiation.cc:407] Connection header received
0619 16:36:04.765770 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:04.765776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:04.765835 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:04.765896 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:04.766279 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.766999 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.768522 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:04.768924 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:04.769073 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:04.769132 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:04.769338 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 16:36:04.769410 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":61,"threads_started":1}
I20260619 16:36:04.770236 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1fed4b2b9d497ebe98f682df9493da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" is_pre_election: true
I20260619 16:36:04.770274 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e1fed4b2b9d497ebe98f682df9493da" is_pre_election: true
I20260619 16:36:04.770432 7102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1b02b63e5c431d9fd6bb2972bc301a in term 1.
I20260619 16:36:04.770433 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e1fed4b2b9d497ebe98f682df9493da in term 1.
I20260619 16:36:04.770757 7077 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1fed4b2b9d497ebe98f682df9493da, 9c1b02b63e5c431d9fd6bb2972bc301a; no voters: 47d0f9f98d954ad985e316c65f7a5cac
I20260619 16:36:04.770757 7259 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1fed4b2b9d497ebe98f682df9493da, 9c1b02b63e5c431d9fd6bb2972bc301a; no voters: 47d0f9f98d954ad985e316c65f7a5cac
I20260619 16:36:04.770965 7796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:36:04.771024 7796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47d0f9f98d954ad985e316c65f7a5cac)
I20260619 16:36:04.771015 7795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 16:36:04.771064 7796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:36:04.771066 7795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47d0f9f98d954ad985e316c65f7a5cac)
I20260619 16:36:04.771112 7795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:36:04.772033 7795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:04.772042 7796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:04.772353 7795 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 2 election: Requested vote from peers 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897), 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61:37975)
I20260619 16:36:04.772387 7796 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [CANDIDATE]: Term 2 election: Requested vote from peers 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897), 9c1b02b63e5c431d9fd6bb2972bc301a (127.0.98.60:33025)
I20260619 16:36:04.772665 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.772440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44706)
0619 16:36:04.772559 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.772606 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:36:04.772729 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e1fed4b2b9d497ebe98f682df9493da"
I20260619 16:36:04.772805 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.772443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44718)
0619 16:36:04.772694 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.772745 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20260619 16:36:04.772784 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1fed4b2b9d497ebe98f682df9493da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a"
I20260619 16:36:04.772868 7102 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9c1b02b63e5c431d9fd6bb2972bc301a in current term 2: Already voted for candidate 5e1fed4b2b9d497ebe98f682df9493da in this term.
I20260619 16:36:04.772938 7285 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e1fed4b2b9d497ebe98f682df9493da in current term 2: Already voted for candidate 9c1b02b63e5c431d9fd6bb2972bc301a in this term.
I20260619 16:36:04.773283 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.773043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44730)
0619 16:36:04.773179 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.773227 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 16:36:04.773429 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:04.773199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44742)
0619 16:36:04.773286 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:04.773358 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 16:36:04.773416 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.773478 7262 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c1b02b63e5c431d9fd6bb2972bc301a; no voters: 47d0f9f98d954ad985e316c65f7a5cac, 5e1fed4b2b9d497ebe98f682df9493da
W20260619 16:36:04.773514 7079 leader_election.cc:334] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:04.773589 7079 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1fed4b2b9d497ebe98f682df9493da; no voters: 47d0f9f98d954ad985e316c65f7a5cac, 9c1b02b63e5c431d9fd6bb2972bc301a
I20260619 16:36:04.773602 7795 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:36:04.773762 7796 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 16:36:05.252918 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:05.252479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:43539)
0619 16:36:05.252755 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:05.252829 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260619 16:36:05.253010 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:05.252549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:40927)
0619 16:36:05.252830 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:05.252883 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":139,"threads_started":1}
I20260619 16:36:05.256482 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:05.256123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:48491)
0619 16:36:05.256358 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:05.256406 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":75,"threads_started":1}
I20260619 16:36:06.253986 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.253542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:40633)
0619 16:36:06.253834 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.253885 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":82,"threads_started":1}
I20260619 16:36:06.253981 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.253592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:41077)
0619 16:36:06.253839 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.253897 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260619 16:36:06.257632 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.257154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:51587)
0619 16:36:06.257456 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.257507 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260619 16:36:06.531590 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:06.531761 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:06.532255 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897), 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61:37975)
I20260619 16:36:06.532819 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e1fed4b2b9d497ebe98f682df9493da" is_pre_election: true
I20260619 16:36:06.532918 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.532409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44984)
0619 16:36:06.532780 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.532838 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":64,"threads_started":1}
I20260619 16:36:06.533000 7102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c1b02b63e5c431d9fd6bb2972bc301a in term 2.
I20260619 16:36:06.533406 7259 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e1fed4b2b9d497ebe98f682df9493da, 9c1b02b63e5c431d9fd6bb2972bc301a; no voters:
I20260619 16:36:06.533576 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 16:36:06.533651 7811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:06.533704 7811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:36:06.533771 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.533555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:44998)
0619 16:36:06.533651 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.533704 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 16:36:06.533886 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:06.534744 7811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:06.535076 7811 leader_election.cc:288] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 3 election: Requested vote from peers 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897), 5e1fed4b2b9d497ebe98f682df9493da (127.0.98.61:37975)
I20260619 16:36:06.535403 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e1fed4b2b9d497ebe98f682df9493da"
I20260619 16:36:06.535511 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.535323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:45012)
0619 16:36:06.535421 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.535464 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 16:36:06.535549 7102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:36:06.536152 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.535964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:45018)
0619 16:36:06.536060 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.536103 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 16:36:06.536314 7262 leader_election.cc:334] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
I20260619 16:36:06.536629 7102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c1b02b63e5c431d9fd6bb2972bc301a in term 3.
I20260619 16:36:06.536890 7259 leader_election.cc:302] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1fed4b2b9d497ebe98f682df9493da, 9c1b02b63e5c431d9fd6bb2972bc301a; no voters: 47d0f9f98d954ad985e316c65f7a5cac
I20260619 16:36:06.537051 7811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:36:06.537247 7811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [term 3 LEADER]: Becoming Leader. State: Replica: 9c1b02b63e5c431d9fd6bb2972bc301a, State: Running, Role: LEADER
I20260619 16:36:06.537524 7811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } }
I20260619 16:36:06.538861 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1b02b63e5c431d9fd6bb2972bc301a. Latest consensus state: current_term: 3 leader_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:06.538978 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:06.539275 7816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:06.541430 7816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:06.541810 7816 catalog_manager.cc:1259] Loaded cluster ID: b9b51045b86c49e285eaf80ec741b0ab
I20260619 16:36:06.541854 7816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:06.542416 7816 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:06.542799 7816 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Loaded TSK: 0
I20260619 16:36:06.543457 7816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:36:06.582372 7275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:06.585668 7275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:06.588653 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.588419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:45034)
0619 16:36:06.588530 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.588589 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:36:06.588815 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [term 3 FOLLOWER]: Refusing update from remote peer 9c1b02b63e5c431d9fd6bb2972bc301a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 16:36:06.589380 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.589145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.0.1:45044)
0619 16:36:06.589226 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.589270 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260619 16:36:06.589417 7811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 16:36:06.589629 7262 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a -> Peer 47d0f9f98d954ad985e316c65f7a5cac (127.0.98.62:36897): Couldn't send request to peer 47d0f9f98d954ad985e316c65f7a5cac. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 16:36:06.590997 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c1b02b63e5c431d9fd6bb2972bc301a. Latest consensus state: current_term: 3 leader_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:06.591148 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:06.592190 7811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:06.592327 7811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:06.592628 7819 mvcc.cc:205] Tried to move back new op lower bound from 7298609015142993920 to 7298609014939963392. Current Snapshot: MvccSnapshot[applied={T|T < 7298609015142993920}]
I20260619 16:36:06.592815 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "47d0f9f98d954ad985e316c65f7a5cac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36897 } } peers { permanent_uuid: "5e1fed4b2b9d497ebe98f682df9493da" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37975 } attrs { promote: false } } peers { permanent_uuid: "9c1b02b63e5c431d9fd6bb2972bc301a" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33025 } attrs { promote: false } } }
I20260619 16:36:06.592958 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1fed4b2b9d497ebe98f682df9493da [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:36:06.603443 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.598622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45481 (local address 127.0.0.1:42176)
0619 16:36:06.598731 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.598746 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:06.598830 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:06.599330 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:06.599334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:06.599354 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:06.599481 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:06.599491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.600897 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.600900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:06.602127 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:06.602136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.602989 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.602992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:06.603102 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.603128 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:36:06.603210 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:36:06.603304 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":44}
I20260619 16:36:06.603478 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.598769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.0.98.3:45481)
0619 16:36:06.599037 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:06.599044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:06.599062 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:36:06.599121 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:06.599124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:06.599178 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:06.599846 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:06.599956 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.600685 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.602279 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.602884 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.602994 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.603061 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:06.603240 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:36:06.603326 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260619 16:36:06.603860 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.598853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.0.1:35792)
0619 16:36:06.599466 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.599482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:06.599588 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:06.600122 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:06.600125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:06.600144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:06.600234 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:06.600241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.601105 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.601109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:06.602545 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:06.602555 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.603475 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.603478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:06.603576 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.603605 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 16:36:06.603675 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:36:06.603740 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":85,"threads_started":1}
I20260619 16:36:06.603896 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.599663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.0.98.1:37615)
0619 16:36:06.599901 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:06.599905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:06.599916 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:36:06.599972 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:06.599974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:06.600010 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:06.600079 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:06.600330 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.600992 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.602680 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.603271 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.603366 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.603420 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:06.603706 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 16:36:06.603776 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260619 16:36:06.604498 7662 tablet_service.cc:1467] Processing CreateTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 (DEFAULT_TABLE table=test-table [id=9979b0b8f5484b9682a2bdb1c50d0e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:06.604791 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f419964ce6994b0f85e6ac87d84f5da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:06.604866 7388 tablet_service.cc:1467] Processing CreateTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 (DEFAULT_TABLE table=test-table [id=9979b0b8f5484b9682a2bdb1c50d0e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:06.605155 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f419964ce6994b0f85e6ac87d84f5da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:06.607051 7829 tablet_bootstrap.cc:492] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Bootstrap starting.
I20260619 16:36:06.608105 7829 tablet_bootstrap.cc:654] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:06.608245 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.599166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39291 (local address 127.0.0.1:59418)
0619 16:36:06.600984 (+ 1818us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.600995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:06.601101 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:06.601762 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:06.601765 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:06.601782 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:06.601858 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:06.601864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.602918 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.602922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:06.604190 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:06.604200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.607853 (+ 3653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.607856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:06.607939 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.607958 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:36:06.608030 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:36:06.608083 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1764,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260619 16:36:06.608330 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.599290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.2:39291)
0619 16:36:06.601309 (+ 2019us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:06.601314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:06.601327 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:06.601407 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:06.601410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:06.601457 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:06.601625 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:06.601988 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.602800 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.604328 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.607705 (+ 3377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.607832 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.607894 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:06.608073 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:36:06.608177 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1935,"thread_start_us":82,"threads_started":1}
I20260619 16:36:06.608829 7830 tablet_bootstrap.cc:492] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb: Bootstrap starting.
I20260619 16:36:06.608951 7525 tablet_service.cc:1467] Processing CreateTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 (DEFAULT_TABLE table=test-table [id=9979b0b8f5484b9682a2bdb1c50d0e74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:06.609211 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f419964ce6994b0f85e6ac87d84f5da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:06.609642 7829 tablet_bootstrap.cc:492] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: No bootstrap required, opened a new log
I20260619 16:36:06.609743 7829 ts_tablet_manager.cc:1397] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:36:06.610121 7829 raft_consensus.cc:348] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.610217 7829 raft_consensus.cc:374] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:06.610241 7829 raft_consensus.cc:729] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a982f53b199477faf9d8df3904c89a2, State: Initialized, Role: FOLLOWER
I20260619 16:36:06.610394 7829 consensus_queue.cc:260] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.610994 7829 ts_tablet_manager.cc:1428] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:36:06.611552 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.611077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:47681)
0619 16:36:06.611425 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.611480 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260619 16:36:06.611938 7830 tablet_bootstrap.cc:654] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:06.612957 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.612567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:47065)
0619 16:36:06.612828 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.612883 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260619 16:36:06.613217 7830 tablet_bootstrap.cc:492] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb: No bootstrap required, opened a new log
I20260619 16:36:06.613292 7830 ts_tablet_manager.cc:1397] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 16:36:06.613723 7833 tablet_bootstrap.cc:492] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Bootstrap starting.
I20260619 16:36:06.613775 7728 heartbeater.cc:503] Master 127.0.98.60:33025 was elected leader, sending a full tablet report...
I20260619 16:36:06.613842 7830 raft_consensus.cc:348] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.613969 7830 raft_consensus.cc:374] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:06.614007 7830 raft_consensus.cc:729] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419a6fb8404345bb876dafd02d76affb, State: Initialized, Role: FOLLOWER
I20260619 16:36:06.614146 7830 consensus_queue.cc:260] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.614527 7830 ts_tablet_manager.cc:1428] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:06.615263 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.615019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:51189)
0619 16:36:06.615127 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.615187 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:36:06.615020 7833 tablet_bootstrap.cc:654] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:06.616081 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.615764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:54603)
0619 16:36:06.615955 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.616003 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 16:36:06.616241 7454 heartbeater.cc:503] Master 127.0.98.60:33025 was elected leader, sending a full tablet report...
I20260619 16:36:06.616961 7833 tablet_bootstrap.cc:492] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: No bootstrap required, opened a new log
I20260619 16:36:06.617058 7833 ts_tablet_manager.cc:1397] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:36:06.617677 7833 raft_consensus.cc:348] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.617796 7833 raft_consensus.cc:374] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:06.617861 7833 raft_consensus.cc:729] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd289fed93242188e10f130ae14e046, State: Initialized, Role: FOLLOWER
I20260619 16:36:06.617997 7833 consensus_queue.cc:260] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.618350 7833 ts_tablet_manager.cc:1428] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 16:36:06.619279 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.619002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:39653)
0619 16:36:06.619124 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.619185 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:36:06.620280 7591 heartbeater.cc:503] Master 127.0.98.60:33025 was elected leader, sending a full tablet report...
I20260619 16:36:06.813375 7832 raft_consensus.cc:482] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:06.813536 7832 raft_consensus.cc:504] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.813907 7832 leader_election.cc:288] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615), bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:06.814509 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f419964ce6994b0f85e6ac87d84f5da2" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419a6fb8404345bb876dafd02d76affb" is_pre_election: true
I20260619 16:36:06.814509 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f419964ce6994b0f85e6ac87d84f5da2" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd289fed93242188e10f130ae14e046" is_pre_election: true
I20260619 16:36:06.814692 7545 raft_consensus.cc:2454] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 0.
I20260619 16:36:06.814692 7408 raft_consensus.cc:2454] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 0.
I20260619 16:36:06.815061 7617 leader_election.cc:302] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a982f53b199477faf9d8df3904c89a2, bdd289fed93242188e10f130ae14e046; no voters:
I20260619 16:36:06.815248 7832 raft_consensus.cc:2781] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:06.815310 7832 raft_consensus.cc:482] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:06.815351 7832 raft_consensus.cc:3037] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:06.816464 7832 raft_consensus.cc:504] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.816833 7832 leader_election.cc:288] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 election: Requested vote from peers 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615), bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291)
I20260619 16:36:06.817235 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f419964ce6994b0f85e6ac87d84f5da2" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419a6fb8404345bb876dafd02d76affb"
I20260619 16:36:06.817263 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f419964ce6994b0f85e6ac87d84f5da2" candidate_uuid: "1a982f53b199477faf9d8df3904c89a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd289fed93242188e10f130ae14e046"
I20260619 16:36:06.817401 7545 raft_consensus.cc:3037] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:06.817396 7408 raft_consensus.cc:3037] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:06.818588 7545 raft_consensus.cc:2454] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 1.
I20260619 16:36:06.818588 7408 raft_consensus.cc:2454] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a982f53b199477faf9d8df3904c89a2 in term 1.
I20260619 16:36:06.818886 7617 leader_election.cc:302] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a982f53b199477faf9d8df3904c89a2, bdd289fed93242188e10f130ae14e046; no voters:
I20260619 16:36:06.819120 7832 raft_consensus.cc:2781] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:06.819371 7832 raft_consensus.cc:686] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 1 LEADER]: Becoming Leader. State: Replica: 1a982f53b199477faf9d8df3904c89a2, State: Running, Role: LEADER
I20260619 16:36:06.819559 7832 consensus_queue.cc:237] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } }
I20260619 16:36:06.820276 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.820025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:52887)
0619 16:36:06.820134 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.820200 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 16:36:06.821146 7275 catalog_manager.cc:5495] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a982f53b199477faf9d8df3904c89a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a982f53b199477faf9d8df3904c89a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 } health_report { overall_health: UNKNOWN } } }
I20260619 16:36:06.837139 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.834168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.0.98.3:45481)
0619 16:36:06.834288 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:06.834292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:06.834360 (+ 68us) server_negotiation.cc:407] Connection header received
0619 16:36:06.834471 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:06.834473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:06.834547 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:06.834613 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:06.834849 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.835336 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.836219 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:06.836425 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:06.836536 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:06.836796 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:06.836855 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:06.836908 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:06.837005 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 16:36:06.837044 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:36:06.839324 7545 raft_consensus.cc:1261] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Refusing update from remote peer 1a982f53b199477faf9d8df3904c89a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:06.839324 7408 raft_consensus.cc:1261] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Refusing update from remote peer 1a982f53b199477faf9d8df3904c89a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:06.839771 7832 consensus_queue.cc:1035] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419a6fb8404345bb876dafd02d76affb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:06.839926 7838 consensus_queue.cc:1035] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd289fed93242188e10f130ae14e046" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:06.841624 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.841167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:60681)
0619 16:36:06.841477 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.841546 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260619 16:36:06.842604 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.841912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:34391)
0619 16:36:06.842481 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.842536 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260619 16:36:06.844920 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.844271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:35457)
0619 16:36:06.844795 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.844851 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456}
I20260619 16:36:06.845255 7842 mvcc.cc:205] Tried to move back new op lower bound from 7298609016169111552 to 7298609016094130176. Current Snapshot: MvccSnapshot[applied={T|T < 7298609016169111552}]
I20260619 16:36:06.845969 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.845484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.1:40551)
0619 16:36:06.845845 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.845904 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
I20260619 16:36:06.847616 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.847204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.3:57005)
0619 16:36:06.847489 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.847542 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260619 16:36:06.848109 7843 mvcc.cc:205] Tried to move back new op lower bound from 7298609016169111552 to 7298609016094130176. Current Snapshot: MvccSnapshot[applied={T|T < 7298609016169111552}]
I20260619 16:36:06.848739 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:06.848496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36897 (local address 127.0.98.2:33781)
0619 16:36:06.848622 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.848672 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:36:06.849002 7275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:06.849140 7275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:06.853050 7275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c1b02b63e5c431d9fd6bb2972bc301a: Sending DeleteTablet for 3 replicas of tablet f419964ce6994b0f85e6ac87d84f5da2
I20260619 16:36:06.853812 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42176
I20260619 16:36:06.853835 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35792
I20260619 16:36:06.854005 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59418
I20260619 16:36:06.853812 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet f419964ce6994b0f85e6ac87d84f5da2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:06 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:42176
I20260619 16:36:06.854192 7850 tablet_replica.cc:330] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb: stopping tablet replica
I20260619 16:36:06.854305 7852 tablet_replica.cc:330] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: stopping tablet replica
I20260619 16:36:06.854347 7850 raft_consensus.cc:2229] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:06.854486 7850 raft_consensus.cc:2258] T f419964ce6994b0f85e6ac87d84f5da2 P 419a6fb8404345bb876dafd02d76affb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:06.854532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 7237
I20260619 16:36:06.855235 7852 raft_consensus.cc:2229] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:06.855340 7852 raft_consensus.cc:2258] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:06.855379 7851 tablet_replica.cc:330] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: stopping tablet replica
I20260619 16:36:06.855528 7851 raft_consensus.cc:2229] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:06.855710 7851 raft_consensus.cc:2258] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:06.855963 7852 ts_tablet_manager.cc:1905] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:06.856398 7851 ts_tablet_manager.cc:1905] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:06.858059 7852 ts_tablet_manager.cc:1918] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:06.858150 7852 log.cc:1196] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/f419964ce6994b0f85e6ac87d84f5da2
I20260619 16:36:06.858361 7852 ts_tablet_manager.cc:1939] T f419964ce6994b0f85e6ac87d84f5da2 P bdd289fed93242188e10f130ae14e046: Deleting consensus metadata
I20260619 16:36:06.858994 7261 catalog_manager.cc:4841] TS bdd289fed93242188e10f130ae14e046 (127.0.98.2:39291): tablet f419964ce6994b0f85e6ac87d84f5da2 (table test-table [id=9979b0b8f5484b9682a2bdb1c50d0e74]) successfully deleted
I20260619 16:36:06.859638 7851 ts_tablet_manager.cc:1918] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:06.859707 7851 log.cc:1196] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/f419964ce6994b0f85e6ac87d84f5da2
I20260619 16:36:06.860004 7851 ts_tablet_manager.cc:1939] T f419964ce6994b0f85e6ac87d84f5da2 P 1a982f53b199477faf9d8df3904c89a2: Deleting consensus metadata
I20260619 16:36:06.860682 7261 catalog_manager.cc:4841] TS 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3:45481): tablet f419964ce6994b0f85e6ac87d84f5da2 (table test-table [id=9979b0b8f5484b9682a2bdb1c50d0e74]) successfully deleted
W20260619 16:36:06.861053 7261 catalog_manager.cc:4808] TS 1a982f53b199477faf9d8df3904c89a2 (127.0.98.3:45481): delete failed for tablet f419964ce6994b0f85e6ac87d84f5da2 because the tablet was not found. No further retry: Not found: Tablet not found: f419964ce6994b0f85e6ac87d84f5da2
W20260619 16:36:06.864054 7826 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:06.863766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.0.1:35800)
0619 16:36:06.863878 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:06.863937 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37615: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 16:36:06.864241 7261 catalog_manager.cc:4584] TS 419a6fb8404345bb876dafd02d76affb (127.0.98.1:37615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f419964ce6994b0f85e6ac87d84f5da2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37615: connect: Connection reset by peer (error 104)
I20260619 16:36:06.864750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 7462
I20260619 16:36:06.872228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 7599
I20260619 16:36:06.879565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 6955
I20260619 16:36:06.885187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 7060
2026-06-19T16:36:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7225 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:36:06 dist-test-slave-0f28 krb5kdc[7855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:36:06 dist-test-slave-0f28 krb5kdc[7855](info): set up 2 sockets
Jun 19 16:36:06 dist-test-slave-0f28 krb5kdc[7855](info): commencing operation
krb5kdc: starting...
W20260619 16:36:07.949913 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 16:36:08.067864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33435
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:08.174189 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:08.174439 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:08.174479 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:08.178431 7868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:08.178478 7868 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 16:36:08.178494 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:08.178512 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:08.178527 7868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:08.178542 7868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:08.183732 7868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33435
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:08.184708 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:08.185747 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:08.194123 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:08.194156 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.194147 7868 server_base.cc:1034] running on GCE node
W20260619 16:36:08.194120 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.194926 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:08.195950 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:08.197119 7868 hybrid_clock.cc:648] HybridClock initialized: now 1781886968197089 us; error 43 us; skew 500 ppm
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:08.199765 7868 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 16:36:08.200584 7868 webserver.cc:460] Webserver started at http://127.0.98.62:33441/ using document root <none> and password file <none>
I20260619 16:36:08.201117 7868 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:08.201207 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:08.201471 7868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:08.203459 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "0517e2342f87449085fe393ee919b770"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "a022c7a18bfcdc91d6c55cd317df067c"
server_key_iv: "1f556c65ad60f641c4c231dd2992f74a"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.203985 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "0517e2342f87449085fe393ee919b770"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "a022c7a18bfcdc91d6c55cd317df067c"
server_key_iv: "1f556c65ad60f641c4c231dd2992f74a"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.207437 7868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:36:08.209690 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.210850 7868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:08.210989 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "0517e2342f87449085fe393ee919b770"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "a022c7a18bfcdc91d6c55cd317df067c"
server_key_iv: "1f556c65ad60f641c4c231dd2992f74a"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.211102 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:08.221879 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:08.224506 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:08.224689 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:08.232400 7868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33435
I20260619 16:36:08.232424 7945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33435 every 8 connection(s)
I20260619 16:36:08.233287 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:08.235909 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:08.240950 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Bootstrap starting.
I20260619 16:36:08.243115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 7868
I20260619 16:36:08.243216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:36:08.243345 7946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:08.243530 392 external_mini_cluster.cc:1428] Setting key 8a08ed8ba1d6f6bbfcef76f93df52c56
I20260619 16:36:08.244180 7946 log.cc:824] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:08.246081 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: No bootstrap required, opened a new log
I20260619 16:36:08.248809 7946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0517e2342f87449085fe393ee919b770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260619 16:36:08.249029 7946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:08.249119 7946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0517e2342f87449085fe393ee919b770, State: Initialized, Role: FOLLOWER
I20260619 16:36:08.249564 7946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0517e2342f87449085fe393ee919b770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260619 16:36:08.249706 7946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:08.249791 7946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:08.249886 7946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:08.250680 7946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0517e2342f87449085fe393ee919b770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
I20260619 16:36:08.251035 7946 leader_election.cc:302] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0517e2342f87449085fe393ee919b770; no voters:
I20260619 16:36:08.251362 7946 leader_election.cc:288] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:36:08.251451 7951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:08.251667 7951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [term 1 LEADER]: Becoming Leader. State: Replica: 0517e2342f87449085fe393ee919b770, State: Running, Role: LEADER
I20260619 16:36:08.251999 7951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0517e2342f87449085fe393ee919b770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } }
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:08.252633 7946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:08.253932 7952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0517e2342f87449085fe393ee919b770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0517e2342f87449085fe393ee919b770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } } }
I20260619 16:36:08.254027 7952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:08.253472 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0517e2342f87449085fe393ee919b770. Latest consensus state: current_term: 1 leader_uuid: "0517e2342f87449085fe393ee919b770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0517e2342f87449085fe393ee919b770" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33435 } } }
I20260619 16:36:08.254598 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:08.255131 7965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:08.258085 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.244563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.0.98.62:33435)
0619 16:36:08.245011 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.245026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.245060 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:36:08.245726 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.245742 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.246040 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.246399 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.247152 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.248496 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.249520 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.250144 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.250311 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.253193 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.253200 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.253215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.253242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.255361 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.255883 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.255893 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.255897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.255995 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.256276 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.256281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.256282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.256636 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.256853 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.257069 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.257394 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
I20260619 16:36:08.258114 7965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:08.263216 7965 catalog_manager.cc:1347] Generated new cluster ID: 638b870cd2c3480c92f702fb8e4598e9
I20260619 16:36:08.263288 7965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:08.272663 7965 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:36:08.273569 7965 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:08.280674 7965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Generated new TSK 0
I20260619 16:36:08.281224 7965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 16:36:08.314332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:08.438226 7973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:08.438457 7973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:08.438498 7973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:08.442430 7973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:08.442483 7973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:08.442520 7973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:36:08.442553 7973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:08.447504 7973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:08.448431 7973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:08.449391 7973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:08.458760 7981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:08.458825 7979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.458920 7973 server_base.cc:1034] running on GCE node
W20260619 16:36:08.458765 7978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.459358 7973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:08.460001 7973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:08.461194 7973 hybrid_clock.cc:648] HybridClock initialized: now 1781886968461166 us; error 41 us; skew 500 ppm
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:08.463929 7973 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:36:08.464738 7973 webserver.cc:460] Webserver started at http://127.0.98.1:40501/ using document root <none> and password file <none>
I20260619 16:36:08.465281 7973 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:08.465435 7973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:08.465660 7973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:08.467603 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b7b0131ae35465cbbe193d39164d81e"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "581dcdd531de18655635df7446084541"
server_key_iv: "a9418f40b6e00e1e8206088f1e34379a"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.468158 7973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b7b0131ae35465cbbe193d39164d81e"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "581dcdd531de18655635df7446084541"
server_key_iv: "a9418f40b6e00e1e8206088f1e34379a"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.471839 7973 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 16:36:08.474216 7988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.475453 7973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:36:08.475612 7973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "0b7b0131ae35465cbbe193d39164d81e"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "581dcdd531de18655635df7446084541"
server_key_iv: "a9418f40b6e00e1e8206088f1e34379a"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.475725 7973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:08.502624 7973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:08.505601 7973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:08.505800 7973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:08.507169 7973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:08.507242 7973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.507318 7973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:08.507361 7973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:08.517997 7973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40765
I20260619 16:36:08.518149 8103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40765 every 8 connection(s)
I20260619 16:36:08.518957 7973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:08.520895 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.508059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50759 (local address 127.0.98.62:33435)
0619 16:36:08.508179 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.508183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.509158 (+ 975us) server_negotiation.cc:407] Connection header received
0619 16:36:08.510160 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.510163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.510218 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.510282 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.511961 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.512835 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.514325 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.514748 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.514869 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.517385 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.517391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.517398 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.517429 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.518980 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.519483 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.519489 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.519494 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.519583 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.519958 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.519961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.519962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.520125 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.520205 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.520602 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.520724 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:36:08.520922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 7973
I20260619 16:36:08.521030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:36:08.521349 392 external_mini_cluster.cc:1428] Setting key 7237e7ff1bf4324f7c1ff55e6c226f6b
I20260619 16:36:08.521621 7997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.508578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.1:50759)
0619 16:36:08.509007 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:08.509056 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:08.509900 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:08.510713 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:08.510724 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:08.511160 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:08.511788 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.511806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.513009 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.513013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.514207 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.514220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.514936 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.514940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.515136 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.515709 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:08.515724 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:08.517195 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:08.519102 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.519108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.519120 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.519389 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.519688 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.519690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.519692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.519873 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.520216 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:08.520220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:08.520413 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 16:36:08.520559 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:36:08.520770 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260619 16:36:08.523042 8104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33435
I20260619 16:36:08.523296 8104 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:08.523890 8104 heartbeater.cc:511] Master 127.0.98.62:33435 requested a full tablet report, sending...
I20260619 16:36:08.525394 7900 ts_manager.cc:194] Registered new tserver with Master: 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765)
I20260619 16:36:08.526723 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50759
I20260619 16:36:08.534062 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.527238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42973 (local address 127.0.98.62:33435)
0619 16:36:08.527362 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.527366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.527380 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:08.527417 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.527423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.527471 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.527550 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.527947 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.528656 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.529480 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.529873 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.530001 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.530972 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.530974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.530976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.531017 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.532396 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.532883 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.532886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.532887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.532961 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.533250 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.533252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.533253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.533470 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.533547 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.533848 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.533932 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:36:08.534199 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.525649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.1:42973)
0619 16:36:08.525936 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:08.525954 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:08.526044 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:08.527590 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:08.527595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:08.527752 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:08.527839 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.527844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.528780 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.528783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.529314 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.529383 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.529988 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.529991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.530058 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.530545 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:08.530556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:08.530847 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:08.532516 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.532522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.532527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.532788 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.533057 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.533061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.533064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.533159 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.533570 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:08.533575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:08.533678 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 16:36:08.533803 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:36:08.533956 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 16:36:08.551858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:08.655774 8113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:08.655992 8113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:08.656032 8113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:08.659950 8113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:08.659998 8113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:08.660035 8113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:36:08.660126 8113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:08.665040 8113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:08.665993 8113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:08.666932 8113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:08.676043 8119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.676252 8113 server_base.cc:1034] running on GCE node
W20260619 16:36:08.676748 8121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:08.676772 8118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.677155 8113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:08.677754 8113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:08.678907 8113 hybrid_clock.cc:648] HybridClock initialized: now 1781886968678892 us; error 29 us; skew 500 ppm
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:08.681588 8113 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:36:08.682360 8113 webserver.cc:460] Webserver started at http://127.0.98.2:32815/ using document root <none> and password file <none>
I20260619 16:36:08.682852 8113 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:08.682907 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:08.683060 8113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:08.684873 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c5322dd2f774488b22c10cf4fa9180b"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "be8ccbc351645c0d24c055a057cb6a34"
server_key_iv: "cda6f8f39fd489dd83e98abcd8c13f0e"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.685405 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c5322dd2f774488b22c10cf4fa9180b"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "be8ccbc351645c0d24c055a057cb6a34"
server_key_iv: "cda6f8f39fd489dd83e98abcd8c13f0e"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.688864 8113 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:36:08.691171 8128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.692332 8113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:08.692420 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "9c5322dd2f774488b22c10cf4fa9180b"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "be8ccbc351645c0d24c055a057cb6a34"
server_key_iv: "cda6f8f39fd489dd83e98abcd8c13f0e"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.692543 8113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:08.706007 8113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:08.708734 8113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:08.708922 8113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:08.710271 8113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:08.710345 8113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.710409 8113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:08.710469 8113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:08.721411 8113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34831
I20260619 16:36:08.721491 8243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34831 every 8 connection(s)
I20260619 16:36:08.722389 8113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:08.723799 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.711204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38819 (local address 127.0.98.62:33435)
0619 16:36:08.711326 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.711329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.712286 (+ 957us) server_negotiation.cc:407] Connection header received
0619 16:36:08.713528 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.713531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.713592 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.713696 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.714965 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.715567 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.716697 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.716928 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.717042 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.719836 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.719840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.719842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.719869 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.721589 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.722052 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.722058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.722063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.722158 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.722578 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.722582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.722583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.722829 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.722933 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.723581 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.723683 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:36:08.724572 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.711691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.2:38819)
0619 16:36:08.712146 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:08.712185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:08.713175 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:08.713852 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:08.713858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:08.714223 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:08.714829 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.714845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.715698 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.715702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.716568 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.716577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.717104 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.717109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.717298 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.717937 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:08.717973 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:08.719671 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:08.721691 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.721696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.721705 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.721947 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.722276 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.722279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.722280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.722407 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.722992 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:08.722998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:08.723304 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 16:36:08.723524 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 16:36:08.723768 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260619 16:36:08.725739 8244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33435
I20260619 16:36:08.725867 8244 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:08.726387 8244 heartbeater.cc:511] Master 127.0.98.62:33435 requested a full tablet report, sending...
I20260619 16:36:08.727458 7900 ts_manager.cc:194] Registered new tserver with Master: 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831)
I20260619 16:36:08.727705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 8113
I20260619 16:36:08.727825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:36:08.728173 392 external_mini_cluster.cc:1428] Setting key 94a6e1e97b4e76270eea7f8a7de1401e
I20260619 16:36:08.728209 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38819
I20260619 16:36:08.736562 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.727995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.2:51305)
0619 16:36:08.728259 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:08.728275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:08.728358 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:08.729179 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:08.729181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:08.729443 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:08.729561 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.729569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.730503 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.730509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.731294 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.731302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.731934 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.731937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.732001 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.732506 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:08.732516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:08.732801 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:08.734987 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.734990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.734991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.735209 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.735508 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.735510 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.735511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.735587 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.736075 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:08.736077 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:08.736155 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 16:36:08.736250 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:36:08.736363 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260619 16:36:08.736646 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.728079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51305 (local address 127.0.98.62:33435)
0619 16:36:08.728653 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.728659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.728673 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:08.728706 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.728709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.728748 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.728827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.729688 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.730367 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.731409 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.731813 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.731928 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.732935 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.732948 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.732954 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.732985 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.734865 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.735312 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.735318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.735322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.735413 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.735680 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.735686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.735689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.735967 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.736083 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.736338 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.736457 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:36:08.756556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33435
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:08.866334 8253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:08.866765 8253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:08.866876 8253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:08.874558 8253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:08.874650 8253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:08.874740 8253 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 16:36:08.874822 8253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:08.883059 8253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:08.884528 8253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:08.886081 8253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:08.896039 8261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:08.896093 8259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:08.896389 8258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:08.896349 8253 server_base.cc:1034] running on GCE node
I20260619 16:36:08.896797 8253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:08.897524 8253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:08.898722 8253 hybrid_clock.cc:648] HybridClock initialized: now 1781886968898708 us; error 26 us; skew 500 ppm
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:08.901688 8253 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:36:08.902560 8253 webserver.cc:460] Webserver started at http://127.0.98.3:38303/ using document root <none> and password file <none>
I20260619 16:36:08.903131 8253 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:08.903223 8253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:08.903486 8253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:08.905639 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "54bb6e2de4124bedba4b9639638aa8ad"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "6dd93ce073146ebf0b9f2984b16a1e8c"
server_key_iv: "4408a13b4901d8b46177313d3bbba262"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.906219 8253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54bb6e2de4124bedba4b9639638aa8ad"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "6dd93ce073146ebf0b9f2984b16a1e8c"
server_key_iv: "4408a13b4901d8b46177313d3bbba262"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.910084 8253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 16:36:08.912642 8268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.913928 8253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 16:36:08.914075 8253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "54bb6e2de4124bedba4b9639638aa8ad"
format_stamp: "Formatted at 2026-06-19 16:36:08 on dist-test-slave-0f28"
server_key: "6dd93ce073146ebf0b9f2984b16a1e8c"
server_key_iv: "4408a13b4901d8b46177313d3bbba262"
server_key_version: "encryptionkey@0"
I20260619 16:36:08.914212 8253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:08.930567 8253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:08.933833 8253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:08.934032 8253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:08.935545 8253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:08.935593 8253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:08.935632 8253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:08.935688 8253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:08.946431 8253 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38731
I20260619 16:36:08.946472 8383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38731 every 8 connection(s)
I20260619 16:36:08.947505 8253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:08.950011 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.936540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45573 (local address 127.0.98.62:33435)
0619 16:36:08.936674 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.936678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.937847 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 16:36:08.939032 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.939037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.939108 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.939221 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.940822 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.941754 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.943287 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.943609 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.943739 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.946527 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.946534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.946537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.946565 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.948136 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.948659 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.948662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.948663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.948739 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.949009 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.949011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.949012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.949174 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.949263 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.949787 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.949883 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:36:08.950766 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.937103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.3:45573)
0619 16:36:08.937687 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:08.937727 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:08.938770 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:08.939411 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:08.939490 (+ 79us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:08.939939 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:08.940631 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.940655 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.941890 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.941894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.943148 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.943158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.943757 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.943760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.943894 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.944570 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:08.944592 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:08.946312 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:08.948271 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.948276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.948289 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.948556 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.948805 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.948807 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.948809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.948926 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.949273 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:08.949277 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:08.949549 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 16:36:08.949729 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 16:36:08.949953 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":156,"threads_started":1}
I20260619 16:36:08.951910 8384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33435
I20260619 16:36:08.952067 8384 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:08.952610 8384 heartbeater.cc:511] Master 127.0.98.62:33435 requested a full tablet report, sending...
I20260619 16:36:08.952605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 8253
I20260619 16:36:08.952742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:36:08.953071 392 external_mini_cluster.cc:1428] Setting key 47f316ca593e449521b503ae9b4034a6
I20260619 16:36:08.953758 7898 ts_manager.cc:194] Registered new tserver with Master: 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:08.954559 7898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45573
I20260619 16:36:08.955212 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:36:08.961936 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.954244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.98.3:39681)
0619 16:36:08.954531 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:08.954546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:08.954635 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:08.955202 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:08.955205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:08.955394 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:08.955486 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.955491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.956260 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.956263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.956826 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:08.956833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.957435 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.957437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:08.957507 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.958023 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:08.958034 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:08.958490 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:08.960210 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.960214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.960216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.960497 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.960800 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:08.960802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:08.960804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:08.960878 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:08.961311 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:08.961313 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:08.961479 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 16:36:08.961567 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:36:08.961666 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260619 16:36:08.961971 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.954864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39681 (local address 127.0.98.62:33435)
0619 16:36:08.954995 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.955002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.955017 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:36:08.955051 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.955053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.955091 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.955185 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.955641 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.956140 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.956943 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.957311 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.957441 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.958647 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.958649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.958651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.958671 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.960101 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.960597 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.960609 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.960614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.960701 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.960968 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.960974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.960979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.961211 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.961335 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.961659 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.961788 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:36:08.978695 7950 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:08.972920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.62:33435)
0619 16:36:08.973058 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.973063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.973074 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:36:08.973113 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.973115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.973147 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.973233 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.973973 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.974471 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.975301 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.975517 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.975624 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.978592 (+ 2968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42786: BlockingRecv error: recv got EOF from 127.0.0.1:42786 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:36:08 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:08.988217 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:08.980545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.0.98.62:33435)
0619 16:36:08.980655 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:08.980658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:08.980790 (+ 132us) server_negotiation.cc:407] Connection header received
0619 16:36:08.980865 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:08.980867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:08.980900 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:08.980981 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:08.981599 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.982090 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.982919 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:08.983154 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:08.983236 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:08.985385 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:08.985387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:08.985389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:08.985409 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:08.986925 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.987315 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.987317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.987318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.987387 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:08.987602 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:08.987603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:08.987604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:08.987761 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:08.987857 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:08.988027 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 16:36:08.988107 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:36:08.990978 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:08.994036 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:09.007977 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.002117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40765 (local address 127.0.0.1:42296)
0619 16:36:09.002874 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.002937 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.003062 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.004017 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.004022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.004041 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.004137 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.004141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.005363 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.005367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.006668 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.006676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.007510 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.007513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.007627 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.007667 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.007763 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.007861 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":77,"threads_started":1}
I20260619 16:36:09.008503 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.002311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.0.98.1:40765)
0619 16:36:09.003562 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.003576 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.003627 (+ 51us) server_negotiation.cc:407] Connection header received
0619 16:36:09.003706 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.003711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.003874 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.004003 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.004251 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.005226 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.006785 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.007395 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.007508 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.007662 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.008325 (+ 663us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.008399 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":76,"threads_started":1}
I20260619 16:36:09.008893 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.002747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.0.1:48734)
0619 16:36:09.004178 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.004189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.004301 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.004753 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.004756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.004775 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.004865 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.004871 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.006065 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.006068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.007233 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.007243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.008481 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.008485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.008584 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.008618 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.008685 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.008750 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1380,"thread_start_us":110,"threads_started":1}
I20260619 16:36:09.009265 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.002247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38731 (local address 127.0.0.1:43966)
0619 16:36:09.003012 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.003023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.003089 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.004922 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.004925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.004943 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.005026 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.005034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.006585 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.006589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.007651 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.007659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.008875 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.008878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.008982 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.009007 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.009081 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.009142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":116,"threads_started":1}
I20260619 16:36:09.009279 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.003264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.0.98.2:34831)
0619 16:36:09.003583 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.003597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.004289 (+ 692us) server_negotiation.cc:407] Connection header received
0619 16:36:09.004413 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.004419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.004610 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.004754 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.004991 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.005939 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.007372 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.008159 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.008264 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.008407 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.009058 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.009147 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
I20260619 16:36:09.009724 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.002880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.0.98.3:38731)
0619 16:36:09.004543 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.004553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.004573 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:36:09.004638 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.004643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.004805 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.004947 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.005150 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.006473 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.007753 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.008572 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.008682 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.008820 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.009503 (+ 683us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.009592 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1573,"thread_start_us":100,"threads_started":1}
I20260619 16:36:09.011425 8038 tablet_service.cc:1467] Processing CreateTablet for tablet e476712782ca46708ba26354b35ebd53 (DEFAULT_TABLE table=test-table [id=2354f8e0e87c478981535b07f24daed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.011690 8178 tablet_service.cc:1467] Processing CreateTablet for tablet e476712782ca46708ba26354b35ebd53 (DEFAULT_TABLE table=test-table [id=2354f8e0e87c478981535b07f24daed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.012082 8318 tablet_service.cc:1467] Processing CreateTablet for tablet e476712782ca46708ba26354b35ebd53 (DEFAULT_TABLE table=test-table [id=2354f8e0e87c478981535b07f24daed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.012516 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e476712782ca46708ba26354b35ebd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.012602 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e476712782ca46708ba26354b35ebd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.012956 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e476712782ca46708ba26354b35ebd53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.019148 8412 tablet_bootstrap.cc:492] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Bootstrap starting.
I20260619 16:36:09.020521 8413 tablet_bootstrap.cc:492] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Bootstrap starting.
I20260619 16:36:09.021253 8414 tablet_bootstrap.cc:492] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Bootstrap starting.
I20260619 16:36:09.021497 8412 tablet_bootstrap.cc:654] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.022351 8412 log.cc:824] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:09.022442 8413 tablet_bootstrap.cc:654] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.023243 8413 log.cc:824] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:09.023535 8414 tablet_bootstrap.cc:654] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.024343 8414 log.cc:824] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:09.024472 8412 tablet_bootstrap.cc:492] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: No bootstrap required, opened a new log
I20260619 16:36:09.024644 8412 ts_tablet_manager.cc:1397] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260619 16:36:09.025039 8413 tablet_bootstrap.cc:492] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: No bootstrap required, opened a new log
I20260619 16:36:09.025234 8413 ts_tablet_manager.cc:1397] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:36:09.026131 8414 tablet_bootstrap.cc:492] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: No bootstrap required, opened a new log
I20260619 16:36:09.026304 8414 ts_tablet_manager.cc:1397] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:36:09.028008 8413 raft_consensus.cc:348] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.028242 8413 raft_consensus.cc:374] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.028294 8413 raft_consensus.cc:729] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b0131ae35465cbbe193d39164d81e, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.028573 8412 raft_consensus.cc:348] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.028715 8413 consensus_queue.cc:260] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.028820 8412 raft_consensus.cc:374] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.028874 8412 raft_consensus.cc:729] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.029278 8104 heartbeater.cc:503] Master 127.0.98.62:33435 was elected leader, sending a full tablet report...
I20260619 16:36:09.029258 8412 consensus_queue.cc:260] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.029928 8244 heartbeater.cc:503] Master 127.0.98.62:33435 was elected leader, sending a full tablet report...
I20260619 16:36:09.030088 8414 raft_consensus.cc:348] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.030300 8412 ts_tablet_manager.cc:1428] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 16:36:09.030303 8414 raft_consensus.cc:374] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.030359 8414 raft_consensus.cc:729] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.030732 8414 consensus_queue.cc:260] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.031317 8384 heartbeater.cc:503] Master 127.0.98.62:33435 was elected leader, sending a full tablet report...
I20260619 16:36:09.032785 8413 ts_tablet_manager.cc:1428] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:36:09.033560 8419 raft_consensus.cc:482] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:09.033691 8419 raft_consensus.cc:504] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.033766 8414 ts_tablet_manager.cc:1428] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:36:09.034789 8419 leader_election.cc:288] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:09.038453 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.035183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35057 (local address 127.0.98.3:38731)
0619 16:36:09.035281 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.035286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.035299 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:09.035429 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.035433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.035477 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.035576 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.035818 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.036308 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.037519 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.037919 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.038039 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.038097 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.038268 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.038333 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:36:09.038450 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.035078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38731 (local address 127.0.98.2:35057)
0619 16:36:09.035217 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.035235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.035318 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.035586 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.035589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.035606 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.035716 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.035726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.036422 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.036427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.037393 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.037404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.038037 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.038041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.038138 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.038159 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.038256 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.038333 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:36:09.038978 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e476712782ca46708ba26354b35ebd53" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bb6e2de4124bedba4b9639638aa8ad" is_pre_election: true
I20260619 16:36:09.039207 8338 raft_consensus.cc:2454] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 0.
I20260619 16:36:09.039638 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.035417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40765 (local address 127.0.98.2:55843)
0619 16:36:09.035962 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.035973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.036077 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.036324 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.036327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.036343 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.036471 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.036476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.037181 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.037184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.038356 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.038365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.039291 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.039297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.039397 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.039414 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.039474 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.039522 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260619 16:36:09.039707 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.035515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55843 (local address 127.0.98.1:40765)
0619 16:36:09.035627 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.035632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.036045 (+ 413us) server_negotiation.cc:407] Connection header received
0619 16:36:09.036182 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.036184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.036222 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.036304 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.036572 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.037060 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.038480 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.039031 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.039153 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.039207 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.039519 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.039596 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:36:09.039680 8129 leader_election.cc:302] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54bb6e2de4124bedba4b9639638aa8ad, 9c5322dd2f774488b22c10cf4fa9180b; no voters:
I20260619 16:36:09.039948 8419 raft_consensus.cc:2781] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:09.040035 8419 raft_consensus.cc:482] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:09.040067 8419 raft_consensus.cc:3037] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.040264 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e476712782ca46708ba26354b35ebd53" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e" is_pre_election: true
I20260619 16:36:09.040503 8058 raft_consensus.cc:2454] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 0.
I20260619 16:36:09.041224 8419 raft_consensus.cc:504] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.041599 8419 leader_election.cc:288] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 election: Requested vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:09.041896 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e476712782ca46708ba26354b35ebd53" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e"
I20260619 16:36:09.041958 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e476712782ca46708ba26354b35ebd53" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bb6e2de4124bedba4b9639638aa8ad"
I20260619 16:36:09.041982 8058 raft_consensus.cc:3037] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.042074 8338 raft_consensus.cc:3037] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.042914 8058 raft_consensus.cc:2454] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 1.
I20260619 16:36:09.042994 8338 raft_consensus.cc:2454] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 1.
I20260619 16:36:09.043287 8129 leader_election.cc:302] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54bb6e2de4124bedba4b9639638aa8ad, 9c5322dd2f774488b22c10cf4fa9180b; no voters:
I20260619 16:36:09.043438 8419 raft_consensus.cc:2781] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:09.043617 8419 raft_consensus.cc:686] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 LEADER]: Becoming Leader. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Running, Role: LEADER
I20260619 16:36:09.043896 8419 consensus_queue.cc:237] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } }
I20260619 16:36:09.046617 7898 catalog_manager.cc:5495] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9c5322dd2f774488b22c10cf4fa9180b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: HEALTHY } } }
I20260619 16:36:09.058176 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.054760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.0.98.2:34831)
0619 16:36:09.054873 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.054878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.054924 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:36:09.055108 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.055111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.055157 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.055256 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:09.055498 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.055981 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.056820 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.057042 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.057152 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.057452 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.057814 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.057931 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.058004 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.058051 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:36:09.063144 8058 raft_consensus.cc:1261] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Refusing update from remote peer 9c5322dd2f774488b22c10cf4fa9180b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.063490 8338 raft_consensus.cc:1261] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Refusing update from remote peer 9c5322dd2f774488b22c10cf4fa9180b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.063843 8422 consensus_queue.cc:1035] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.064043 8419 consensus_queue.cc:1035] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.072296 8427 mvcc.cc:205] Tried to move back new op lower bound from 7298609025276628992 to 7298609025206169600. Current Snapshot: MvccSnapshot[applied={T|T < 7298609025276628992}]
I20260619 16:36:09.078135 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:09.078315 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:09.080735 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Sending DeleteTablet for 3 replicas of tablet e476712782ca46708ba26354b35ebd53
I20260619 16:36:09.081440 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet e476712782ca46708ba26354b35ebd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42296
I20260619 16:36:09.081468 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet e476712782ca46708ba26354b35ebd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260619 16:36:09.081897 8434 tablet_replica.cc:330] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: stopping tablet replica
I20260619 16:36:09.081918 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet e476712782ca46708ba26354b35ebd53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48734
I20260619 16:36:09.082096 8435 tablet_replica.cc:330] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: stopping tablet replica
I20260619 16:36:09.082206 8434 raft_consensus.cc:2229] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:09.082445 8435 raft_consensus.cc:2229] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:09.082540 8434 raft_consensus.cc:2258] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.083000 8435 raft_consensus.cc:2258] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.083009 7898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:09.083380 7898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:09.083899 8434 ts_tablet_manager.cc:1905] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.085160 8435 ts_tablet_manager.cc:1905] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.086624 8434 ts_tablet_manager.cc:1918] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:09.086848 8434 log.cc:1196] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/e476712782ca46708ba26354b35ebd53
I20260619 16:36:09.087218 8434 ts_tablet_manager.cc:1939] T e476712782ca46708ba26354b35ebd53 P 0b7b0131ae35465cbbe193d39164d81e: Deleting consensus metadata
I20260619 16:36:09.088201 8318 tablet_service.cc:1467] Processing CreateTablet for tablet 4e55449866a94d2a9f4b3d4a9b8741d2 (DEFAULT_TABLE table=test-table [id=c2f6a3999805484ab07353a64bbd769b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.088434 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e55449866a94d2a9f4b3d4a9b8741d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.088423 7887 catalog_manager.cc:4841] TS 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765): tablet e476712782ca46708ba26354b35ebd53 (table test-table [id=2354f8e0e87c478981535b07f24daed5]) successfully deleted
I20260619 16:36:09.088477 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 4e55449866a94d2a9f4b3d4a9b8741d2 (DEFAULT_TABLE table=test-table [id=c2f6a3999805484ab07353a64bbd769b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.088706 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e55449866a94d2a9f4b3d4a9b8741d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.088744 8038 tablet_service.cc:1467] Processing CreateTablet for tablet 4e55449866a94d2a9f4b3d4a9b8741d2 (DEFAULT_TABLE table=test-table [id=c2f6a3999805484ab07353a64bbd769b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.088968 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e55449866a94d2a9f4b3d4a9b8741d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.090495 8414 tablet_bootstrap.cc:492] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Bootstrap starting.
I20260619 16:36:09.090982 8412 tablet_bootstrap.cc:492] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Bootstrap starting.
I20260619 16:36:09.091003 8413 tablet_bootstrap.cc:492] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Bootstrap starting.
I20260619 16:36:09.091439 8414 tablet_bootstrap.cc:654] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.091951 8413 tablet_bootstrap.cc:654] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.091962 8412 tablet_bootstrap.cc:654] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.092897 8414 tablet_bootstrap.cc:492] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: No bootstrap required, opened a new log
I20260619 16:36:09.092981 8414 ts_tablet_manager.cc:1397] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 16:36:09.093432 8414 raft_consensus.cc:348] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.093542 8414 raft_consensus.cc:374] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.093571 8414 raft_consensus.cc:729] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.093690 8414 consensus_queue.cc:260] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.093779 8412 tablet_bootstrap.cc:492] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: No bootstrap required, opened a new log
I20260619 16:36:09.093853 8412 ts_tablet_manager.cc:1397] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260619 16:36:09.093918 8414 ts_tablet_manager.cc:1428] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.093966 8413 tablet_bootstrap.cc:492] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: No bootstrap required, opened a new log
I20260619 16:36:09.094038 8413 ts_tablet_manager.cc:1397] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:36:09.094302 8412 raft_consensus.cc:348] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.094399 8412 raft_consensus.cc:374] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.094435 8412 raft_consensus.cc:729] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.094403 8413 raft_consensus.cc:348] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.094515 8413 raft_consensus.cc:374] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.094545 8413 raft_consensus.cc:729] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b0131ae35465cbbe193d39164d81e, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.094554 8412 consensus_queue.cc:260] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.094650 8413 consensus_queue.cc:260] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.094805 8412 ts_tablet_manager.cc:1428] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.094870 8413 ts_tablet_manager.cc:1428] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.096093 8435 ts_tablet_manager.cc:1918] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:09.096199 8435 log.cc:1196] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/e476712782ca46708ba26354b35ebd53
I20260619 16:36:09.096534 8435 ts_tablet_manager.cc:1939] T e476712782ca46708ba26354b35ebd53 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting consensus metadata
I20260619 16:36:09.097673 8436 tablet_replica.cc:330] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: stopping tablet replica
I20260619 16:36:09.097680 7884 catalog_manager.cc:4841] TS 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731): tablet e476712782ca46708ba26354b35ebd53 (table test-table [id=2354f8e0e87c478981535b07f24daed5]) successfully deleted
I20260619 16:36:09.097944 8436 raft_consensus.cc:2229] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:09.098413 8436 raft_consensus.cc:2258] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.099596 8436 ts_tablet_manager.cc:1905] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.101956 8436 ts_tablet_manager.cc:1918] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:09.102046 8436 log.cc:1196] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/e476712782ca46708ba26354b35ebd53
I20260619 16:36:09.102321 8436 ts_tablet_manager.cc:1939] T e476712782ca46708ba26354b35ebd53 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting consensus metadata
I20260619 16:36:09.103498 7885 catalog_manager.cc:4841] TS 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831): tablet e476712782ca46708ba26354b35ebd53 (table test-table [id=2354f8e0e87c478981535b07f24daed5]) successfully deleted
I20260619 16:36:09.117372 8422 raft_consensus.cc:482] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:09.117475 8422 raft_consensus.cc:504] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.117857 8422 leader_election.cc:288] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:09.118211 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e55449866a94d2a9f4b3d4a9b8741d2" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e" is_pre_election: true
I20260619 16:36:09.118362 8058 raft_consensus.cc:2454] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 0.
I20260619 16:36:09.118450 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e55449866a94d2a9f4b3d4a9b8741d2" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bb6e2de4124bedba4b9639638aa8ad" is_pre_election: true
I20260619 16:36:09.118580 8338 raft_consensus.cc:2454] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 0.
I20260619 16:36:09.118701 8132 leader_election.cc:302] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 9c5322dd2f774488b22c10cf4fa9180b; no voters:
I20260619 16:36:09.118891 8422 raft_consensus.cc:2781] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:09.118943 8422 raft_consensus.cc:482] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:09.118990 8422 raft_consensus.cc:3037] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.120054 8422 raft_consensus.cc:504] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.120373 8422 leader_election.cc:288] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 election: Requested vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:09.120659 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e55449866a94d2a9f4b3d4a9b8741d2" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bb6e2de4124bedba4b9639638aa8ad"
I20260619 16:36:09.120697 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e55449866a94d2a9f4b3d4a9b8741d2" candidate_uuid: "9c5322dd2f774488b22c10cf4fa9180b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e"
I20260619 16:36:09.120792 8058 raft_consensus.cc:3037] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.120795 8338 raft_consensus.cc:3037] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.121727 8338 raft_consensus.cc:2454] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 1.
I20260619 16:36:09.122309 8058 raft_consensus.cc:2454] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c5322dd2f774488b22c10cf4fa9180b in term 1.
I20260619 16:36:09.122614 8132 leader_election.cc:302] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 9c5322dd2f774488b22c10cf4fa9180b; no voters:
I20260619 16:36:09.122771 8422 raft_consensus.cc:2781] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:09.122852 8422 raft_consensus.cc:686] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 LEADER]: Becoming Leader. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Running, Role: LEADER
I20260619 16:36:09.123003 8422 consensus_queue.cc:237] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.124468 7899 catalog_manager.cc:5495] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9c5322dd2f774488b22c10cf4fa9180b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } health_report { overall_health: UNKNOWN } } }
Jun 19 16:36:09 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886968, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:09.163674 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.155132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33435 (local address 127.0.0.1:42804)
0619 16:36:09.155387 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.155420 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.155545 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.155798 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.155801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.156196 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:09.156279 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.156285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.157026 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.157032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.157674 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.157682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.158104 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.158107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.158178 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.158773 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:09.158784 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:09.160364 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:09.162148 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:09.162158 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:09.162165 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:09.162383 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:09.162652 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:09.162654 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:09.162656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:09.162734 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:09.163098 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:09.163107 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:09.163213 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.163322 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.163474 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260619 16:36:09.163674 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.155181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42804 (local address 127.0.98.62:33435)
0619 16:36:09.155317 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.155322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.155491 (+ 169us) server_negotiation.cc:407] Connection header received
0619 16:36:09.155654 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.155658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.155697 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.155777 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:09.156397 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.156922 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.157787 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.158000 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.158104 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.160531 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:09.160534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:09.160536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:09.160555 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:09.162025 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:09.162478 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:09.162484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:09.162486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:09.162558 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:09.162823 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:09.162826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:09.162827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:09.162991 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:09.163090 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.163373 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.163526 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":51}
I20260619 16:36:09.167423 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42804:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 16:36:09.168133 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:09.172999 8038 tablet_service.cc:1467] Processing CreateTablet for tablet cedb46373e0d4c24b4cd2fbedd94d38c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3370537e34c4757b057ff7505a284f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:36:09.173162 8178 tablet_service.cc:1467] Processing CreateTablet for tablet cedb46373e0d4c24b4cd2fbedd94d38c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3370537e34c4757b057ff7505a284f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:36:09.173441 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedb46373e0d4c24b4cd2fbedd94d38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.173583 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedb46373e0d4c24b4cd2fbedd94d38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.175653 8412 tablet_bootstrap.cc:492] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: Bootstrap starting.
I20260619 16:36:09.176512 8412 tablet_bootstrap.cc:654] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.176508 8318 tablet_service.cc:1467] Processing CreateTablet for tablet cedb46373e0d4c24b4cd2fbedd94d38c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c3370537e34c4757b057ff7505a284f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 16:36:09.176864 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cedb46373e0d4c24b4cd2fbedd94d38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.177573 8412 tablet_bootstrap.cc:492] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: No bootstrap required, opened a new log
I20260619 16:36:09.177680 8412 ts_tablet_manager.cc:1397] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:09.178093 8412 raft_consensus.cc:348] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.178221 8412 raft_consensus.cc:374] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.178259 8412 raft_consensus.cc:729] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.178388 8412 consensus_queue.cc:260] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.178695 8412 ts_tablet_manager.cc:1428] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.178926 8422 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.179042 8422 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.179706 8414 tablet_bootstrap.cc:492] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: Bootstrap starting.
I20260619 16:36:09.180025 8413 tablet_bootstrap.cc:492] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: Bootstrap starting.
I20260619 16:36:09.180713 8414 tablet_bootstrap.cc:654] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.181034 8413 tablet_bootstrap.cc:654] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.182512 8414 tablet_bootstrap.cc:492] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: No bootstrap required, opened a new log
I20260619 16:36:09.182613 8414 ts_tablet_manager.cc:1397] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:36:09.183056 8414 raft_consensus.cc:348] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.183171 8414 raft_consensus.cc:374] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.183208 8414 raft_consensus.cc:729] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.183333 8414 consensus_queue.cc:260] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.183588 8414 ts_tablet_manager.cc:1428] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 16:36:09.183609 8420 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.183784 8420 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.184103 8413 tablet_bootstrap.cc:492] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: No bootstrap required, opened a new log
I20260619 16:36:09.184186 8413 ts_tablet_manager.cc:1397] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 16:36:09.184594 8413 raft_consensus.cc:348] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.184723 8413 raft_consensus.cc:374] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.184762 8413 raft_consensus.cc:729] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b0131ae35465cbbe193d39164d81e, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.184888 8413 consensus_queue.cc:260] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.185156 8413 ts_tablet_manager.cc:1428] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.185374 8418 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.185509 8418 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.190642 8420 raft_consensus.cc:482] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:09.190764 8420 raft_consensus.cc:504] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.191855 8420 leader_election.cc:288] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831)
I20260619 16:36:09.196147 8277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.192025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40765 (local address 127.0.98.3:43507)
0619 16:36:09.192147 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.192162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.192248 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.192729 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.192732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.192752 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.192860 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.192869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.193933 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.193939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.194913 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.194924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.195721 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.195724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.195827 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.195851 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.195929 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.196015 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 16:36:09.196240 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.192292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43507 (local address 127.0.98.1:40765)
0619 16:36:09.192426 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.192431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.192445 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:09.192492 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.192496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.192546 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.192618 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.192995 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.193826 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.195047 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.195597 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.195729 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.195795 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.196043 (+ 248us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.196119 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:36:09.196623 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedb46373e0d4c24b4cd2fbedd94d38c" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e" is_pre_election: true
I20260619 16:36:09.196770 8058 raft_consensus.cc:2454] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 0.
I20260619 16:36:09.196924 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.192142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37857 (local address 127.0.98.2:34831)
0619 16:36:09.192258 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.192263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.192621 (+ 358us) server_negotiation.cc:407] Connection header received
0619 16:36:09.192797 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.192800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.192847 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.192920 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.193724 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.194423 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.195621 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.196214 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.196326 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.196415 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.196738 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.196810 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:36:09.197188 8272 leader_election.cc:302] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 54bb6e2de4124bedba4b9639638aa8ad; no voters:
I20260619 16:36:09.197299 8450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.192062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.98.3:37857)
0619 16:36:09.192533 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.192546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.192696 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.193025 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.193028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.193042 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.193595 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.193604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.194545 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.194548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.195510 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.195517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.196482 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.196485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.196598 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.196619 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.197086 (+ 467us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.197160 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":119,"threads_started":1}
I20260619 16:36:09.197464 8420 raft_consensus.cc:2781] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:09.197516 8420 raft_consensus.cc:482] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:09.197556 8420 raft_consensus.cc:3037] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.197814 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedb46373e0d4c24b4cd2fbedd94d38c" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5322dd2f774488b22c10cf4fa9180b" is_pre_election: true
I20260619 16:36:09.198035 8198 raft_consensus.cc:2454] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 0.
W20260619 16:36:09.198684 8385 tablet.cc:2369] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:36:09.198832 8420 raft_consensus.cc:504] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.199137 8420 leader_election.cc:288] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 election: Requested vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831)
I20260619 16:36:09.199429 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedb46373e0d4c24b4cd2fbedd94d38c" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e"
I20260619 16:36:09.199579 8058 raft_consensus.cc:3037] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.199653 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cedb46373e0d4c24b4cd2fbedd94d38c" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5322dd2f774488b22c10cf4fa9180b"
I20260619 16:36:09.199765 8198 raft_consensus.cc:3037] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.200743 8058 raft_consensus.cc:2454] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 1.
I20260619 16:36:09.201035 8272 leader_election.cc:302] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 54bb6e2de4124bedba4b9639638aa8ad; no voters:
I20260619 16:36:09.201215 8420 raft_consensus.cc:2781] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:09.201205 8198 raft_consensus.cc:2454] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 1.
I20260619 16:36:09.201442 8420 raft_consensus.cc:686] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [term 1 LEADER]: Becoming Leader. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Running, Role: LEADER
I20260619 16:36:09.201702 8420 consensus_queue.cc:237] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.202768 8420 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: TxnStatusTablet state changed. Reason: New leader 54bb6e2de4124bedba4b9639638aa8ad. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.202868 8420 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:36:09.204080 7899 catalog_manager.cc:5495] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } health_report { overall_health: HEALTHY } } }
I20260619 16:36:09.205546 8453 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 16:36:09.205695 8453 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260619 16:36:09.223668 8245 tablet.cc:2369] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:36:09.224797 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.219872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38731 (local address 127.0.0.1:43976)
0619 16:36:09.220102 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.220118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.220221 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.220745 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.220748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.220756 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:36:09.220857 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.220863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.221915 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.221918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.222917 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.222927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.223561 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.223564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.223657 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.223709 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.224494 (+ 785us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.224508 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.224569 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.224651 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 16:36:09.225232 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.219953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.0.98.3:38731)
0619 16:36:09.220405 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.220410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.220424 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:09.220488 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.220491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.220549 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.220633 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:09.220986 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.221789 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.223062 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.223393 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.223482 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.223869 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.224350 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.224452 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.225051 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.225105 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378}
I20260619 16:36:09.228300 8058 raft_consensus.cc:1261] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Refusing update from remote peer 54bb6e2de4124bedba4b9639638aa8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.228385 8198 raft_consensus.cc:1261] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Refusing update from remote peer 54bb6e2de4124bedba4b9639638aa8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.228793 8420 consensus_queue.cc:1035] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.229072 8451 consensus_queue.cc:1035] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.230594 8418 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: TxnStatusTablet state changed. Reason: New leader 54bb6e2de4124bedba4b9639638aa8ad. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.230708 8418 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.230772 8422 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: TxnStatusTablet state changed. Reason: New leader 54bb6e2de4124bedba4b9639638aa8ad. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.230890 8422 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.233811 8451 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.233919 8451 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:36:09.235522 8426 mvcc.cc:205] Tried to move back new op lower bound from 7298609025953144832 to 7298609025852854272. Current Snapshot: MvccSnapshot[applied={T|T < 7298609025953144832}]
I20260619 16:36:09.235685 8418 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.235785 8418 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 0b7b0131ae35465cbbe193d39164d81e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.239022 8420 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.239128 8420 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 54bb6e2de4124bedba4b9639638aa8ad: This TxnStatusTablet replica's current role is: LEADER
I20260619 16:36:09.239966 8425 mvcc.cc:205] Tried to move back new op lower bound from 7298609025953144832 to 7298609025852854272. Current Snapshot: MvccSnapshot[applied={T|T < 7298609025953144832}]
I20260619 16:36:09.240154 8422 tablet_replica.cc:439] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } } }
I20260619 16:36:09.240273 8422 tablet_replica.cc:442] T cedb46373e0d4c24b4cd2fbedd94d38c P 9c5322dd2f774488b22c10cf4fa9180b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 16:36:09.257532 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.253389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40519 (local address 127.0.98.3:38731)
0619 16:36:09.253503 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.253508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.253692 (+ 184us) server_negotiation.cc:407] Connection header received
0619 16:36:09.253838 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.253842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.253896 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.253988 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:09.254269 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.255058 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.256286 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.256641 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.256744 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.257033 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.257114 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.257187 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.257344 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.257396 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:36:09.257543 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.253251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38731 (local address 127.0.98.2:40519)
0619 16:36:09.253620 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.253636 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.253739 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.254014 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.254017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.254024 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:36:09.254133 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.254138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.255175 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.255179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.256165 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.256176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.256761 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.256765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.256855 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.256903 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.257221 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.257227 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.257298 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.257402 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":132,"threads_started":1}
I20260619 16:36:09.264693 8058 raft_consensus.cc:1261] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Refusing update from remote peer 9c5322dd2f774488b22c10cf4fa9180b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.264761 8338 raft_consensus.cc:1261] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Refusing update from remote peer 9c5322dd2f774488b22c10cf4fa9180b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.265148 8422 consensus_queue.cc:1035] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.265337 8419 consensus_queue.cc:1035] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:36:09.270146 8105 tablet.cc:2369] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:36:09.287689 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.283773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.98.3:42361)
0619 16:36:09.283917 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.283950 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.284044 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.284314 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.284317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.284325 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:36:09.284435 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.284439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.285461 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.285465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.286463 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.286474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.286918 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.286921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.287024 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.287076 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.287406 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.287412 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.287476 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.287554 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 16:36:09.287693 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.283837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42361 (local address 127.0.98.2:34831)
0619 16:36:09.283947 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.283951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.283997 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:36:09.284153 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.284156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.284202 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.284284 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:09.284536 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.285309 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.286594 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.286806 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.286887 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.287227 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.287307 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.287370 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.287518 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.287577 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:36:09.306805 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:09.306993 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:09.309306 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Sending DeleteTablet for 3 replicas of tablet 4e55449866a94d2a9f4b3d4a9b8741d2
I20260619 16:36:09.309940 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e55449866a94d2a9f4b3d4a9b8741d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42296
I20260619 16:36:09.310072 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e55449866a94d2a9f4b3d4a9b8741d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260619 16:36:09.310119 8434 tablet_replica.cc:330] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: stopping tablet replica
I20260619 16:36:09.310133 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e55449866a94d2a9f4b3d4a9b8741d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48734
I20260619 16:36:09.310245 8435 tablet_replica.cc:330] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: stopping tablet replica
I20260619 16:36:09.310237 8434 raft_consensus.cc:2229] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:09.310285 8436 tablet_replica.cc:330] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: stopping tablet replica
I20260619 16:36:09.310348 8434 raft_consensus.cc:2258] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.310352 8435 raft_consensus.cc:2229] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:09.310385 8436 raft_consensus.cc:2229] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:09.310469 8435 raft_consensus.cc:2258] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.310554 8436 raft_consensus.cc:2258] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.311188 8434 ts_tablet_manager.cc:1905] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.311391 8436 ts_tablet_manager.cc:1905] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.314356 8435 ts_tablet_manager.cc:1905] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.314594 8434 ts_tablet_manager.cc:1918] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:36:09.314666 8434 log.cc:1196] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/4e55449866a94d2a9f4b3d4a9b8741d2
I20260619 16:36:09.314951 8434 ts_tablet_manager.cc:1939] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 0b7b0131ae35465cbbe193d39164d81e: Deleting consensus metadata
I20260619 16:36:09.316881 8435 ts_tablet_manager.cc:1918] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:36:09.316967 8435 log.cc:1196] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/4e55449866a94d2a9f4b3d4a9b8741d2
I20260619 16:36:09.317219 8435 ts_tablet_manager.cc:1939] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting consensus metadata
I20260619 16:36:09.317806 7887 catalog_manager.cc:4841] TS 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765): tablet 4e55449866a94d2a9f4b3d4a9b8741d2 (table test-table [id=c2f6a3999805484ab07353a64bbd769b]) successfully deleted
I20260619 16:36:09.317880 7884 catalog_manager.cc:4841] TS 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731): tablet 4e55449866a94d2a9f4b3d4a9b8741d2 (table test-table [id=c2f6a3999805484ab07353a64bbd769b]) successfully deleted
I20260619 16:36:09.318341 8436 ts_tablet_manager.cc:1918] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:36:09.318418 8436 log.cc:1196] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/4e55449866a94d2a9f4b3d4a9b8741d2
I20260619 16:36:09.318655 8436 ts_tablet_manager.cc:1939] T 4e55449866a94d2a9f4b3d4a9b8741d2 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting consensus metadata
I20260619 16:36:09.319296 7885 catalog_manager.cc:4841] TS 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831): tablet 4e55449866a94d2a9f4b3d4a9b8741d2 (table test-table [id=c2f6a3999805484ab07353a64bbd769b]) successfully deleted
Jun 19 16:36:09 dist-test-slave-0f28 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 16:36:09 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:09 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:36:09.338972 7950 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:09.332047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.0.98.62:33435)
0619 16:36:09.332181 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.332185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.332429 (+ 244us) server_negotiation.cc:407] Connection header received
0619 16:36:09.332532 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.332535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.332578 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.332651 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:09.333505 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.334033 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.335227 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.335447 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.335610 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.338839 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42806: BlockingRecv error: recv got EOF from 127.0.0.1:42806 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 16:36:09 dist-test-slave-0f28 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 16:36:09.349072 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.340501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.0.98.62:33435)
0619 16:36:09.340606 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.340610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.340763 (+ 153us) server_negotiation.cc:407] Connection header received
0619 16:36:09.340899 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.340901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.340939 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.341002 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:09.341718 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.342191 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.343333 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.343528 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.343597 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.346103 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:09.346107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:09.346109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:09.346135 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:09.347504 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:09.347919 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:09.347923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:09.347925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:09.348002 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:09.348273 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:09.348278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:09.348280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:09.348444 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:09.348579 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.348807 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.348920 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:36:09.351601 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:09.352106 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:09.357151 8038 tablet_service.cc:1467] Processing CreateTablet for tablet 04be1107c2c6407298bf4b5b89c06d86 (DEFAULT_TABLE table=test-table [id=3428b295a9154d2a84e8dc2e27cbb742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.357225 8318 tablet_service.cc:1467] Processing CreateTablet for tablet 04be1107c2c6407298bf4b5b89c06d86 (DEFAULT_TABLE table=test-table [id=3428b295a9154d2a84e8dc2e27cbb742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.357247 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 04be1107c2c6407298bf4b5b89c06d86 (DEFAULT_TABLE table=test-table [id=3428b295a9154d2a84e8dc2e27cbb742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.357503 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04be1107c2c6407298bf4b5b89c06d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.357529 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04be1107c2c6407298bf4b5b89c06d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.357529 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04be1107c2c6407298bf4b5b89c06d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.359848 8414 tablet_bootstrap.cc:492] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Bootstrap starting.
I20260619 16:36:09.359848 8412 tablet_bootstrap.cc:492] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Bootstrap starting.
I20260619 16:36:09.360693 8414 tablet_bootstrap.cc:654] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.360693 8412 tablet_bootstrap.cc:654] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.361265 8413 tablet_bootstrap.cc:492] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Bootstrap starting.
I20260619 16:36:09.362011 8414 tablet_bootstrap.cc:492] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: No bootstrap required, opened a new log
I20260619 16:36:09.362084 8414 ts_tablet_manager.cc:1397] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 16:36:09.362236 8413 tablet_bootstrap.cc:654] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.362573 8414 raft_consensus.cc:348] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.362689 8414 raft_consensus.cc:374] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.362727 8414 raft_consensus.cc:729] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.362844 8414 consensus_queue.cc:260] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.363072 8414 ts_tablet_manager.cc:1428] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.365059 8412 tablet_bootstrap.cc:492] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: No bootstrap required, opened a new log
I20260619 16:36:09.365170 8412 ts_tablet_manager.cc:1397] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 16:36:09.365635 8412 raft_consensus.cc:348] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.365749 8412 raft_consensus.cc:374] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.365782 8412 raft_consensus.cc:729] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.365908 8412 consensus_queue.cc:260] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.366178 8412 ts_tablet_manager.cc:1428] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.366703 8413 tablet_bootstrap.cc:492] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: No bootstrap required, opened a new log
I20260619 16:36:09.366784 8413 ts_tablet_manager.cc:1397] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 16:36:09.367247 8413 raft_consensus.cc:348] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.367358 8413 raft_consensus.cc:374] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.367391 8413 raft_consensus.cc:729] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b0131ae35465cbbe193d39164d81e, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.367511 8413 consensus_queue.cc:260] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.367798 8413 ts_tablet_manager.cc:1428] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.558956 8451 raft_consensus.cc:482] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:09.559085 8451 raft_consensus.cc:504] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.559443 8451 leader_election.cc:288] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831)
I20260619 16:36:09.559883 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04be1107c2c6407298bf4b5b89c06d86" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e" is_pre_election: true
I20260619 16:36:09.559883 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04be1107c2c6407298bf4b5b89c06d86" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5322dd2f774488b22c10cf4fa9180b" is_pre_election: true
I20260619 16:36:09.560034 8058 raft_consensus.cc:2454] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 0.
I20260619 16:36:09.560034 8198 raft_consensus.cc:2454] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 0.
I20260619 16:36:09.560375 8272 leader_election.cc:302] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 54bb6e2de4124bedba4b9639638aa8ad; no voters:
I20260619 16:36:09.560539 8451 raft_consensus.cc:2781] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:09.560602 8451 raft_consensus.cc:482] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:09.560648 8451 raft_consensus.cc:3037] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.561677 8451 raft_consensus.cc:504] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.562001 8451 leader_election.cc:288] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 election: Requested vote from peers 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765), 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831)
I20260619 16:36:09.562286 8058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04be1107c2c6407298bf4b5b89c06d86" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7b0131ae35465cbbe193d39164d81e"
I20260619 16:36:09.562433 8058 raft_consensus.cc:3037] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.562404 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04be1107c2c6407298bf4b5b89c06d86" candidate_uuid: "54bb6e2de4124bedba4b9639638aa8ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5322dd2f774488b22c10cf4fa9180b"
I20260619 16:36:09.562579 8198 raft_consensus.cc:3037] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.563546 8058 raft_consensus.cc:2454] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 1.
I20260619 16:36:09.563726 8198 raft_consensus.cc:2454] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54bb6e2de4124bedba4b9639638aa8ad in term 1.
I20260619 16:36:09.563889 8272 leader_election.cc:302] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 54bb6e2de4124bedba4b9639638aa8ad; no voters:
I20260619 16:36:09.564043 8451 raft_consensus.cc:2781] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:09.564157 8451 raft_consensus.cc:686] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 LEADER]: Becoming Leader. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Running, Role: LEADER
I20260619 16:36:09.564317 8451 consensus_queue.cc:237] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.565985 7899 catalog_manager.cc:5495] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3). New cstate: current_term: 1 leader_uuid: "54bb6e2de4124bedba4b9639638aa8ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } health_report { overall_health: HEALTHY } } }
I20260619 16:36:09.599978 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.597167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.0.98.3:38731)
0619 16:36:09.597305 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.597308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.597358 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:36:09.597411 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.597413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.597456 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.597525 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:09.597761 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.598258 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.599066 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.599271 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.599344 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.599609 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.599671 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.599723 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.599830 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.599870 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:36:09.602180 8198 raft_consensus.cc:1261] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Refusing update from remote peer 54bb6e2de4124bedba4b9639638aa8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.602203 8058 raft_consensus.cc:1261] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Refusing update from remote peer 54bb6e2de4124bedba4b9639638aa8ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:09.602581 8420 consensus_queue.cc:1035] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.602718 8451 consensus_queue.cc:1035] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:09.611905 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:09.612018 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:09.613850 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Sending DeleteTablet for 3 replicas of tablet 04be1107c2c6407298bf4b5b89c06d86
I20260619 16:36:09.614341 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet 04be1107c2c6407298bf4b5b89c06d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42296
I20260619 16:36:09.614480 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 04be1107c2c6407298bf4b5b89c06d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48734
I20260619 16:36:09.614513 8434 tablet_replica.cc:330] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: stopping tablet replica
I20260619 16:36:09.614616 8434 raft_consensus.cc:2229] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:09.614636 8436 tablet_replica.cc:330] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: stopping tablet replica
I20260619 16:36:09.614699 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet 04be1107c2c6407298bf4b5b89c06d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260619 16:36:09.614737 8434 raft_consensus.cc:2258] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.614740 8436 raft_consensus.cc:2229] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:09.614847 8436 raft_consensus.cc:2258] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.614856 8435 tablet_replica.cc:330] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: stopping tablet replica
I20260619 16:36:09.614943 8435 raft_consensus.cc:2229] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:09.615198 8435 raft_consensus.cc:2258] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:09.615473 8436 ts_tablet_manager.cc:1905] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.615509 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:09.615897 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:09.615957 8435 ts_tablet_manager.cc:1905] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.617900 8436 ts_tablet_manager.cc:1918] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:09.617975 8436 log.cc:1196] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/04be1107c2c6407298bf4b5b89c06d86
I20260619 16:36:09.618242 8436 ts_tablet_manager.cc:1939] T 04be1107c2c6407298bf4b5b89c06d86 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting consensus metadata
I20260619 16:36:09.618351 8435 ts_tablet_manager.cc:1918] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:09.618438 8435 log.cc:1196] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/04be1107c2c6407298bf4b5b89c06d86
I20260619 16:36:09.618696 8435 ts_tablet_manager.cc:1939] T 04be1107c2c6407298bf4b5b89c06d86 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting consensus metadata
I20260619 16:36:09.619079 7885 catalog_manager.cc:4841] TS 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831): tablet 04be1107c2c6407298bf4b5b89c06d86 (table test-table [id=3428b295a9154d2a84e8dc2e27cbb742]) successfully deleted
I20260619 16:36:09.619304 7884 catalog_manager.cc:4841] TS 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731): tablet 04be1107c2c6407298bf4b5b89c06d86 (table test-table [id=3428b295a9154d2a84e8dc2e27cbb742]) successfully deleted
I20260619 16:36:09.621025 8038 tablet_service.cc:1467] Processing CreateTablet for tablet 792c13ffc9f54fb0ae0022713eb2e703 (DEFAULT_TABLE table=test-table [id=ae3b3eeba6964860b7c5ca46c16a0137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.621268 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 792c13ffc9f54fb0ae0022713eb2e703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.621222 8318 tablet_service.cc:1467] Processing CreateTablet for tablet 792c13ffc9f54fb0ae0022713eb2e703 (DEFAULT_TABLE table=test-table [id=ae3b3eeba6964860b7c5ca46c16a0137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.621476 8318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 792c13ffc9f54fb0ae0022713eb2e703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.623304 8414 tablet_bootstrap.cc:492] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Bootstrap starting.
I20260619 16:36:09.623489 8434 ts_tablet_manager.cc:1905] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:09.623725 8178 tablet_service.cc:1467] Processing CreateTablet for tablet 792c13ffc9f54fb0ae0022713eb2e703 (DEFAULT_TABLE table=test-table [id=ae3b3eeba6964860b7c5ca46c16a0137]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:09.623934 8178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 792c13ffc9f54fb0ae0022713eb2e703. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:09.624078 8414 tablet_bootstrap.cc:654] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.625082 8414 tablet_bootstrap.cc:492] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: No bootstrap required, opened a new log
I20260619 16:36:09.625200 8414 ts_tablet_manager.cc:1397] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:09.625679 8434 ts_tablet_manager.cc:1918] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:09.625751 8434 log.cc:1196] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/04be1107c2c6407298bf4b5b89c06d86
I20260619 16:36:09.625734 8414 raft_consensus.cc:348] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.625847 8414 raft_consensus.cc:374] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.625883 8414 raft_consensus.cc:729] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54bb6e2de4124bedba4b9639638aa8ad, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.625993 8434 ts_tablet_manager.cc:1939] T 04be1107c2c6407298bf4b5b89c06d86 P 0b7b0131ae35465cbbe193d39164d81e: Deleting consensus metadata
I20260619 16:36:09.626001 8414 consensus_queue.cc:260] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.626202 8412 tablet_bootstrap.cc:492] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Bootstrap starting.
I20260619 16:36:09.626282 8414 ts_tablet_manager.cc:1428] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 16:36:09.626983 8412 tablet_bootstrap.cc:654] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.623500 8413 tablet_bootstrap.cc:492] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Bootstrap starting.
I20260619 16:36:09.628005 8412 tablet_bootstrap.cc:492] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: No bootstrap required, opened a new log
I20260619 16:36:09.628090 8412 ts_tablet_manager.cc:1397] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:09.628162 8413 tablet_bootstrap.cc:654] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:09.628387 8412 raft_consensus.cc:348] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.628460 8412 raft_consensus.cc:374] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.628479 8412 raft_consensus.cc:729] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c5322dd2f774488b22c10cf4fa9180b, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.628548 8412 consensus_queue.cc:260] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.628903 8412 ts_tablet_manager.cc:1428] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.630148 7887 catalog_manager.cc:4841] TS 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765): tablet 04be1107c2c6407298bf4b5b89c06d86 (table test-table [id=3428b295a9154d2a84e8dc2e27cbb742]) successfully deleted
I20260619 16:36:09.630389 8413 tablet_bootstrap.cc:492] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: No bootstrap required, opened a new log
I20260619 16:36:09.630453 8413 ts_tablet_manager.cc:1397] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 16:36:09.630816 8413 raft_consensus.cc:348] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.630909 8413 raft_consensus.cc:374] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:09.630939 8413 raft_consensus.cc:729] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7b0131ae35465cbbe193d39164d81e, State: Initialized, Role: FOLLOWER
I20260619 16:36:09.631052 8413 consensus_queue.cc:260] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.631278 8413 ts_tablet_manager.cc:1428] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:36:09.952574 8418 raft_consensus.cc:482] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:09.952756 8418 raft_consensus.cc:504] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.954115 8418 leader_election.cc:288] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831), 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:09.958621 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.954320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34831 (local address 127.0.98.1:46969)
0619 16:36:09.954576 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.954590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.954726 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.955180 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.955182 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.955198 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.955313 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.955321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.956409 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.956413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.957400 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.957410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.958188 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.958192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.958278 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.958304 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.958351 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.958420 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":48,"thread_start_us":94,"threads_started":1}
I20260619 16:36:09.958974 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.954723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37921 (local address 127.0.98.3:38731)
0619 16:36:09.954853 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.954859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.955020 (+ 161us) server_negotiation.cc:407] Connection header received
0619 16:36:09.955129 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.955133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.955182 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.955248 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.955559 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.956267 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.957866 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.958446 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.958544 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.958603 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.958807 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.958871 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:36:09.959270 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.954719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46969 (local address 127.0.98.2:34831)
0619 16:36:09.954962 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.954966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.954982 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:36:09.955034 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.955037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.955084 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.955174 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:09.955424 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.956270 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.957535 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.958055 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.958940 (+ 885us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.958999 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.959052 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.959140 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":57,"threads_started":1}
I20260619 16:36:09.959594 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.954370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38731 (local address 127.0.98.1:37921)
0619 16:36:09.954927 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:09.954938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:09.955046 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:09.955303 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:09.955306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:09.955324 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:09.955444 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.955450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.956376 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.956379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.957742 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:09.957751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.958564 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.958568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:09.958663 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.958687 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:36:09.959145 (+ 458us) client_negotiation.cc:240] Negotiation successful
0619 16:36:09.959226 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":57,"thread_start_us":83,"threads_started":1}
I20260619 16:36:09.959728 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "792c13ffc9f54fb0ae0022713eb2e703" candidate_uuid: "0b7b0131ae35465cbbe193d39164d81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5322dd2f774488b22c10cf4fa9180b" is_pre_election: true
I20260619 16:36:09.959877 8198 raft_consensus.cc:2454] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b0131ae35465cbbe193d39164d81e in term 0.
I20260619 16:36:09.959950 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "792c13ffc9f54fb0ae0022713eb2e703" candidate_uuid: "0b7b0131ae35465cbbe193d39164d81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bb6e2de4124bedba4b9639638aa8ad" is_pre_election: true
I20260619 16:36:09.960072 8338 raft_consensus.cc:2454] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7b0131ae35465cbbe193d39164d81e in term 0.
I20260619 16:36:09.960470 7989 leader_election.cc:302] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 54bb6e2de4124bedba4b9639638aa8ad, 9c5322dd2f774488b22c10cf4fa9180b; no voters:
I20260619 16:36:09.960727 8418 raft_consensus.cc:2781] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:09.960781 8418 raft_consensus.cc:482] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:09.960819 8418 raft_consensus.cc:3037] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.961855 8418 raft_consensus.cc:504] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.962208 8418 leader_election.cc:288] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [CANDIDATE]: Term 1 election: Requested vote from peers 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831), 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731)
I20260619 16:36:09.962491 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "792c13ffc9f54fb0ae0022713eb2e703" candidate_uuid: "0b7b0131ae35465cbbe193d39164d81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c5322dd2f774488b22c10cf4fa9180b"
I20260619 16:36:09.962508 8338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "792c13ffc9f54fb0ae0022713eb2e703" candidate_uuid: "0b7b0131ae35465cbbe193d39164d81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54bb6e2de4124bedba4b9639638aa8ad"
I20260619 16:36:09.962604 8338 raft_consensus.cc:3037] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.962631 8198 raft_consensus.cc:3037] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:09.963492 8198 raft_consensus.cc:2454] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b0131ae35465cbbe193d39164d81e in term 1.
I20260619 16:36:09.963732 8338 raft_consensus.cc:2454] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7b0131ae35465cbbe193d39164d81e in term 1.
I20260619 16:36:09.963798 7990 leader_election.cc:302] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7b0131ae35465cbbe193d39164d81e, 9c5322dd2f774488b22c10cf4fa9180b; no voters:
I20260619 16:36:09.963972 8418 raft_consensus.cc:2781] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:09.964140 8418 raft_consensus.cc:686] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 1 LEADER]: Becoming Leader. State: Replica: 0b7b0131ae35465cbbe193d39164d81e, State: Running, Role: LEADER
I20260619 16:36:09.964367 8418 consensus_queue.cc:237] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } }
I20260619 16:36:09.966565 7900 catalog_manager.cc:5495] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0b7b0131ae35465cbbe193d39164d81e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7b0131ae35465cbbe193d39164d81e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 } health_report { overall_health: UNKNOWN } } }
I20260619 16:36:09.995038 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:09.991215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.0.98.1:40765)
0619 16:36:09.991547 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:09.991555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:09.991575 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:36:09.991635 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:09.991637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:09.991684 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:09.991816 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:09.992051 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.992680 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.993597 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:09.993823 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:09.993941 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:09.994201 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:09.994650 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:09.994780 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:09.994824 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 16:36:09.994873 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":138,"threads_started":1}
I20260619 16:36:10.004576 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:10.001301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48809 (local address 127.0.98.3:38731)
0619 16:36:10.001513 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:10.001519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:10.001633 (+ 114us) server_negotiation.cc:407] Connection header received
0619 16:36:10.001784 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:10.001789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:10.001839 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:10.001935 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:10.002184 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.002669 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.003533 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.003758 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.003868 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:10.004164 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:10.004228 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:10.004309 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:10.004398 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 16:36:10.004444 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 16:36:10.004609 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:10.001163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38731 (local address 127.0.98.1:48809)
0619 16:36:10.001538 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:10.001560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:10.001685 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:10.001945 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:10.001948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:10.001955 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:36:10.002080 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:10.002085 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.002783 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.002787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:10.003404 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:10.003412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.003869 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.003881 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:10.003975 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:10.004015 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:36:10.004317 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:36:10.004326 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:36:10.004388 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:36:10.004458 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260619 16:36:10.010636 8198 raft_consensus.cc:1261] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Refusing update from remote peer 0b7b0131ae35465cbbe193d39164d81e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:10.010694 8338 raft_consensus.cc:1261] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Refusing update from remote peer 0b7b0131ae35465cbbe193d39164d81e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:10.011133 8418 consensus_queue.cc:1035] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c5322dd2f774488b22c10cf4fa9180b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:10.011375 8493 consensus_queue.cc:1035] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [LEADER]: Connected to new peer: Peer: permanent_uuid: "54bb6e2de4124bedba4b9639638aa8ad" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:10.028091 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:10.024862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53897 (local address 127.0.98.1:40765)
0619 16:36:10.024978 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:10.024983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:10.025262 (+ 279us) server_negotiation.cc:407] Connection header received
0619 16:36:10.025396 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:10.025398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:10.025448 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:10.025513 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:10.025674 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.026161 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.027053 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.027332 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.027428 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:10.027669 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:10.027765 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:10.027837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:10.027957 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 16:36:10.028000 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:36:10.028190 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:10.024814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40765 (local address 127.0.98.3:53897)
0619 16:36:10.025140 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:10.025152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:10.025271 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:10.025515 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:10.025518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:10.025524 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 16:36:10.025608 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:10.025612 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.026245 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.026248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:10.026913 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:10.026925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:10.027426 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:10.027430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:10.027514 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:10.027548 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 16:36:10.027858 (+ 310us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 16:36:10.027866 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:36:10.027941 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:36:10.028034 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":77,"threads_started":1}
I20260619 16:36:10.049963 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:10.050125 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:10.052124 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0517e2342f87449085fe393ee919b770: Sending DeleteTablet for 3 replicas of tablet 792c13ffc9f54fb0ae0022713eb2e703
I20260619 16:36:10.052754 8178 tablet_service.cc:1514] Processing DeleteTablet for tablet 792c13ffc9f54fb0ae0022713eb2e703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48734
I20260619 16:36:10.052773 8038 tablet_service.cc:1514] Processing DeleteTablet for tablet 792c13ffc9f54fb0ae0022713eb2e703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42296
I20260619 16:36:10.052860 8318 tablet_service.cc:1514] Processing DeleteTablet for tablet 792c13ffc9f54fb0ae0022713eb2e703 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:10 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260619 16:36:10.052984 8436 tablet_replica.cc:330] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: stopping tablet replica
I20260619 16:36:10.053013 8435 tablet_replica.cc:330] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: stopping tablet replica
I20260619 16:36:10.053117 8436 raft_consensus.cc:2229] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:10.053124 8435 raft_consensus.cc:2229] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:10.053246 8436 raft_consensus.cc:2258] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:10.053256 8435 raft_consensus.cc:2258] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:10.053444 8434 tablet_replica.cc:330] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: stopping tablet replica
I20260619 16:36:10.053573 8434 raft_consensus.cc:2229] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:10.053846 8434 raft_consensus.cc:2258] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:10.054382 8435 ts_tablet_manager.cc:1905] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:10.054600 8434 ts_tablet_manager.cc:1905] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:10.054637 8436 ts_tablet_manager.cc:1905] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:10.056820 8436 ts_tablet_manager.cc:1918] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:36:10.056820 8435 ts_tablet_manager.cc:1918] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:36:10.056888 8436 log.cc:1196] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/792c13ffc9f54fb0ae0022713eb2e703
I20260619 16:36:10.056888 8435 log.cc:1196] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/792c13ffc9f54fb0ae0022713eb2e703
I20260619 16:36:10.056962 8434 ts_tablet_manager.cc:1918] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 16:36:10.057029 8434 log.cc:1196] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/792c13ffc9f54fb0ae0022713eb2e703
I20260619 16:36:10.057142 8435 ts_tablet_manager.cc:1939] T 792c13ffc9f54fb0ae0022713eb2e703 P 54bb6e2de4124bedba4b9639638aa8ad: Deleting consensus metadata
I20260619 16:36:10.057299 8434 ts_tablet_manager.cc:1939] T 792c13ffc9f54fb0ae0022713eb2e703 P 0b7b0131ae35465cbbe193d39164d81e: Deleting consensus metadata
I20260619 16:36:10.057593 8436 ts_tablet_manager.cc:1939] T 792c13ffc9f54fb0ae0022713eb2e703 P 9c5322dd2f774488b22c10cf4fa9180b: Deleting consensus metadata
I20260619 16:36:10.057782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 7973
I20260619 16:36:10.058889 7887 catalog_manager.cc:4841] TS 0b7b0131ae35465cbbe193d39164d81e (127.0.98.1:40765): tablet 792c13ffc9f54fb0ae0022713eb2e703 (table test-table [id=ae3b3eeba6964860b7c5ca46c16a0137]) successfully deleted
I20260619 16:36:10.059865 7885 catalog_manager.cc:4841] TS 9c5322dd2f774488b22c10cf4fa9180b (127.0.98.2:34831): tablet 792c13ffc9f54fb0ae0022713eb2e703 (table test-table [id=ae3b3eeba6964860b7c5ca46c16a0137]) successfully deleted
I20260619 16:36:10.060577 7884 catalog_manager.cc:4841] TS 54bb6e2de4124bedba4b9639638aa8ad (127.0.98.3:38731): tablet 792c13ffc9f54fb0ae0022713eb2e703 (table test-table [id=ae3b3eeba6964860b7c5ca46c16a0137]) successfully deleted
I20260619 16:36:10.065266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 8113
I20260619 16:36:10.071774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 8253
I20260619 16:36:10.078684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 7868
2026-06-19T16:36:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3192 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:36:10 dist-test-slave-0f28 krb5kdc[8508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:36:10 dist-test-slave-0f28 krb5kdc[8508](info): set up 2 sockets
Jun 19 16:36:10 dist-test-slave-0f28 krb5kdc[8508](info): commencing operation
krb5kdc: starting...
W20260619 16:36:11.126048 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:36:11 dist-test-slave-0f28 krb5kdc[8508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:36:11.244833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40141
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:11.351046 8521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:11.351276 8521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:11.351313 8521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:11.355207 8521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:11.355252 8521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:11.355271 8521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:11.355288 8521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:11.355302 8521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:11.360235 8521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:11.361125 8521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:11.362020 8521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:11.369956 8526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:11.369956 8529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:11.369948 8527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:11.370185 8521 server_base.cc:1034] running on GCE node
I20260619 16:36:11.370766 8521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:11.371769 8521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:11.372939 8521 hybrid_clock.cc:648] HybridClock initialized: now 1781886971372908 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T16:36:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1302 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:36:11 dist-test-slave-0f28 krb5kdc[8534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:36:11 dist-test-slave-0f28 krb5kdc[8534](info): set up 2 sockets
Jun 19 16:36:11 dist-test-slave-0f28 krb5kdc[8534](info): commencing operation
krb5kdc: starting...
W20260619 16:36:12.426107 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:36:12 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:36:12.549222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36445
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:12.654502 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:12.654811 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:12.654884 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:12.658802 8547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:12.658886 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:12.658921 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:12.658958 8547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:12.658989 8547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:12.663980 8547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:12.664922 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:12.665879 8547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:12.673988 8552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:12.674049 8555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:12.674034 8547 server_base.cc:1034] running on GCE node
W20260619 16:36:12.673996 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:12.674782 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:12.675853 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:12.677060 8547 hybrid_clock.cc:648] HybridClock initialized: now 1781886972677045 us; error 35 us; skew 500 ppm
Jun 19 16:36:12 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:12.683223 8547 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:36:12.684199 8547 webserver.cc:460] Webserver started at http://127.0.98.62:34255/ using document root <none> and password file <none>
I20260619 16:36:12.684778 8547 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:12.684876 8547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:12.685112 8547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:12.687284 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "f1808f95ae4e445cb7a240b8d9914608"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "4f766c169251cfd30d86e4e30381e0ce"
server_key_iv: "9b39aecb12b1777418bdaaf055583510"
server_key_version: "encryptionkey@0"
I20260619 16:36:12.687868 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1808f95ae4e445cb7a240b8d9914608"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "4f766c169251cfd30d86e4e30381e0ce"
server_key_iv: "9b39aecb12b1777418bdaaf055583510"
server_key_version: "encryptionkey@0"
I20260619 16:36:12.691651 8547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:36:12.694123 8562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:12.695480 8547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:12.695581 8547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "f1808f95ae4e445cb7a240b8d9914608"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "4f766c169251cfd30d86e4e30381e0ce"
server_key_iv: "9b39aecb12b1777418bdaaf055583510"
server_key_version: "encryptionkey@0"
I20260619 16:36:12.695664 8547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:12.724565 8547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:12.727347 8547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:12.727528 8547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:12.734442 8547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36445
I20260619 16:36:12.734465 8614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36445 every 8 connection(s)
I20260619 16:36:12.735347 8547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:12.737895 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:12.743129 8615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Bootstrap starting.
I20260619 16:36:12.744875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 8547
I20260619 16:36:12.745016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:36:12.745283 392 external_mini_cluster.cc:1428] Setting key 655c463cb87be5f927accec929abcae4
I20260619 16:36:12.745635 8615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:12.746415 8615 log.cc:824] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:12.748222 8615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: No bootstrap required, opened a new log
I20260619 16:36:12.751555 8615 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:12.751777 8615 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:12.751858 8615 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1808f95ae4e445cb7a240b8d9914608, State: Initialized, Role: FOLLOWER
I20260619 16:36:12.752310 8615 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:12.752480 8615 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:12.752568 8615 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:12.752664 8615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:12.753720 8615 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
Jun 19 16:36:12 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:36:12.754091 8615 leader_election.cc:302] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1808f95ae4e445cb7a240b8d9914608; no voters:
I20260619 16:36:12.754419 8615 leader_election.cc:288] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:36:12.754494 8620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:12.754721 8620 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 LEADER]: Becoming Leader. State: Replica: f1808f95ae4e445cb7a240b8d9914608, State: Running, Role: LEADER
I20260619 16:36:12.755060 8620 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:12.755695 8615 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:12.756215 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1808f95ae4e445cb7a240b8d9914608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } } }
I20260619 16:36:12.756980 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:12.756865 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1808f95ae4e445cb7a240b8d9914608. Latest consensus state: current_term: 1 leader_uuid: "f1808f95ae4e445cb7a240b8d9914608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } } }
I20260619 16:36:12.757265 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:12.758061 8629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:12.759655 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:12.746387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45212 (local address 127.0.98.62:36445)
0619 16:36:12.746847 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:12.746867 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:12.746912 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:36:12.747664 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:12.747681 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:12.748006 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:12.748341 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:12.749099 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:12.750112 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:12.751002 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:12.751469 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:12.751636 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:12.754584 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:12.754593 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:12.754610 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:12.754641 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:12.756695 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:12.757201 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:12.757206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:12.757212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:12.757363 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:12.757697 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:12.757702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:12.757704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:12.758202 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:12.758473 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:12.758625 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 16:36:12.758908 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
I20260619 16:36:12.761251 8629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:12.767205 8629 catalog_manager.cc:1347] Generated new cluster ID: 81f01b60498c4b9c9e3695334175b352
I20260619 16:36:12.767288 8629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:12.775403 8629 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:36:12.776257 8629 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:12.783340 8629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Generated new TSK 0
I20260619 16:36:12.784071 8629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:36:12.816542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36445
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:12.927346 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:12.927595 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:12.927639 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:12.931787 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:12.931838 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:12.931924 8641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:12.937067 8641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:12.938104 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:12.939093 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:12.948345 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:12.948365 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:12.948385 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:12.948613 8641 server_base.cc:1034] running on GCE node
I20260619 16:36:12.949074 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:12.949752 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:12.950913 8641 hybrid_clock.cc:648] HybridClock initialized: now 1781886972950902 us; error 25 us; skew 500 ppm
Jun 19 16:36:12 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:12.953685 8641 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:36:12.954507 8641 webserver.cc:460] Webserver started at http://127.0.98.1:33773/ using document root <none> and password file <none>
I20260619 16:36:12.954998 8641 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:12.955054 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:12.955209 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:12.957291 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "32bd78d2576a4e0ea3999ee8aa8a19ab"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "c8271d431a766fd9fe36a50a3e9c8938"
server_key_iv: "68a272441adf3d0e304808be84de9858"
server_key_version: "encryptionkey@0"
I20260619 16:36:12.957847 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32bd78d2576a4e0ea3999ee8aa8a19ab"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "c8271d431a766fd9fe36a50a3e9c8938"
server_key_iv: "68a272441adf3d0e304808be84de9858"
server_key_version: "encryptionkey@0"
I20260619 16:36:12.961282 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:36:12.963570 8656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:12.964740 8641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:12.964869 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "32bd78d2576a4e0ea3999ee8aa8a19ab"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "c8271d431a766fd9fe36a50a3e9c8938"
server_key_iv: "68a272441adf3d0e304808be84de9858"
server_key_version: "encryptionkey@0"
I20260619 16:36:12.964972 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:12.984521 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:12.987389 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:12.987624 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:12.988183 8641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:12.988962 8641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:12.989034 8641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:12.989099 8641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:12.989149 8641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:12.998198 8641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41549
I20260619 16:36:12.998234 8769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41549 every 8 connection(s)
I20260619 16:36:12.999074 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:13.002202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 8641
I20260619 16:36:13.002338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:36:13.002856 392 external_mini_cluster.cc:1428] Setting key e20d3769305c45f3d41c8f2014b6a312
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:36:13.014568 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.000779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50451 (local address 127.0.98.62:36445)
0619 16:36:13.000910 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:13.000914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:13.001797 (+ 883us) server_negotiation.cc:407] Connection header received
0619 16:36:13.003987 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:13.003990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:13.004040 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:13.004123 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:13.005708 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.006495 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.007616 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.008039 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.008164 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.010911 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:13.010917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:13.010921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:13.010950 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:13.012566 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:13.013080 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:13.013083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:13.013085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:13.013162 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:13.013518 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:13.013542 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:13.013545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:13.013739 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:13.013817 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:13.014251 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 16:36:13.014401 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:36:13.015314 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.001138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:50451)
0619 16:36:13.001639 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:13.001688 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:13.003784 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:13.004302 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:13.004329 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:13.004821 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:13.005545 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.005568 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.006652 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.006655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.007481 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.007489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.008177 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.008180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.008393 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.009020 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:13.009035 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:13.010742 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:13.012710 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:13.012719 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:13.012730 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:13.012960 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:13.013244 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:13.013246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:13.013248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:13.013428 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:13.013831 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:13.013836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:13.014042 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 16:36:13.014190 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 16:36:13.014456 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":165,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:36:13.016669 8770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36445
I20260619 16:36:13.016927 8770 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:13.017581 8770 heartbeater.cc:511] Master 127.0.98.62:36445 requested a full tablet report, sending...
I20260619 16:36:13.019233 8579 ts_manager.cc:194] Registered new tserver with Master: 32bd78d2576a4e0ea3999ee8aa8a19ab (127.0.98.1:41549)
I20260619 16:36:13.020917 8579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:36:13.032632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36445
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:13.138608 8775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:13.138842 8775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:13.138883 8775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:13.142747 8775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:13.142789 8775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:13.142843 8775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:13.148165 8775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:13.149020 8775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:13.149950 8775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:13.159240 8783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:13.159300 8775 server_base.cc:1034] running on GCE node
W20260619 16:36:13.159372 8781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:13.159567 8780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:13.159979 8775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:13.160596 8775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:13.161772 8775 hybrid_clock.cc:648] HybridClock initialized: now 1781886973161764 us; error 29 us; skew 500 ppm
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:13.164302 8775 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:36:13.165091 8775 webserver.cc:460] Webserver started at http://127.0.98.2:35787/ using document root <none> and password file <none>
I20260619 16:36:13.165678 8775 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:13.165768 8775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:13.165992 8775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:13.167932 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "d79b9879be2a4b84b5cc610fea69e67b"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "9d28f6bf1500c0e0d3967b1554697cdf"
server_key_iv: "4c59911eb21c40c40f25f40fd610771d"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.168571 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d79b9879be2a4b84b5cc610fea69e67b"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "9d28f6bf1500c0e0d3967b1554697cdf"
server_key_iv: "4c59911eb21c40c40f25f40fd610771d"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.172137 8775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:36:13.174541 8790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.175643 8775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:36:13.175786 8775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "d79b9879be2a4b84b5cc610fea69e67b"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "9d28f6bf1500c0e0d3967b1554697cdf"
server_key_iv: "4c59911eb21c40c40f25f40fd610771d"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.175894 8775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:13.192027 8775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:13.195102 8775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:13.195309 8775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:13.195993 8775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:13.196770 8775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:13.196841 8775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.196909 8775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:13.196961 8775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.206056 8775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45841
I20260619 16:36:13.206079 8903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45841 every 8 connection(s)
I20260619 16:36:13.206899 8775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:13.208468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 8775
I20260619 16:36:13.208608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:36:13.208892 392 external_mini_cluster.cc:1428] Setting key b702dc953f2aeacaf9bc513f7e4356f5
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:36:13.221458 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.208764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33817 (local address 127.0.98.62:36445)
0619 16:36:13.208897 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:13.208900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:13.209898 (+ 998us) server_negotiation.cc:407] Connection header received
0619 16:36:13.211287 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:13.211290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:13.211342 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:13.211413 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:13.212508 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.213036 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.214820 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.215044 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.215131 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.217895 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:13.217898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:13.217900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:13.217921 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:13.219321 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:13.219881 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:13.219885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:13.219887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:13.219971 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:13.220299 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:13.220302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:13.220304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:13.220475 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:13.220673 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:13.221161 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 16:36:13.221275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:36:13.222159 8906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.209238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:33817)
0619 16:36:13.209745 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:13.209795 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:13.211076 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:13.211574 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:13.211580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:13.211907 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:13.212363 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.212376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.213164 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.213170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.214698 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.214711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.215198 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.215205 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.215463 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.216079 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:13.216105 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:13.217729 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:13.219458 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:13.219463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:13.219476 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:13.219779 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:13.220061 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:13.220065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:13.220067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:13.220198 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:13.220640 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:13.220646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:13.220887 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 16:36:13.221062 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 16:36:13.221383 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":140,"threads_started":1}
I20260619 16:36:13.223659 8904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36445
I20260619 16:36:13.223882 8904 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:13.224329 8904 heartbeater.cc:511] Master 127.0.98.62:36445 requested a full tablet report, sending...
I20260619 16:36:13.225309 8579 ts_manager.cc:194] Registered new tserver with Master: d79b9879be2a4b84b5cc610fea69e67b (127.0.98.2:45841)
I20260619 16:36:13.225908 8579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:36:13.236621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36445
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:13.344663 8909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:13.344897 8909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:13.344936 8909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:13.348942 8909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:13.348991 8909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:13.349073 8909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:13.354235 8909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:13.355196 8909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:13.356155 8909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:13.365137 8917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:13.365187 8915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:13.365155 8914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:13.365311 8909 server_base.cc:1034] running on GCE node
I20260619 16:36:13.365785 8909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:13.366384 8909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:13.367600 8909 hybrid_clock.cc:648] HybridClock initialized: now 1781886973367581 us; error 26 us; skew 500 ppm
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:13.370499 8909 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:36:13.371338 8909 webserver.cc:460] Webserver started at http://127.0.98.3:35961/ using document root <none> and password file <none>
I20260619 16:36:13.371861 8909 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:13.371922 8909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:13.372151 8909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:13.374118 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "c4f254371db14ad9835ce8d3d7035f30"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "e6dd3f63858ea9a4972db4e481bf1ad2"
server_key_iv: "14fb8535ae6a62eae0ee92059430598b"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.374634 8909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c4f254371db14ad9835ce8d3d7035f30"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "e6dd3f63858ea9a4972db4e481bf1ad2"
server_key_iv: "14fb8535ae6a62eae0ee92059430598b"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.378083 8909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:36:13.380492 8924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.381760 8909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:13.381868 8909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "c4f254371db14ad9835ce8d3d7035f30"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "e6dd3f63858ea9a4972db4e481bf1ad2"
server_key_iv: "14fb8535ae6a62eae0ee92059430598b"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.381975 8909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:13.391178 8909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:13.393975 8909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:13.394152 8909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:13.394682 8909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:13.395419 8909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:13.395463 8909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.395527 8909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:13.395567 8909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.404668 8909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43337
I20260619 16:36:13.404690 9037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43337 every 8 connection(s)
I20260619 16:36:13.405592 8909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:13.412171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 8909
I20260619 16:36:13.412281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:36:13.412529 392 external_mini_cluster.cc:1428] Setting key ccf71549afa4838ebd079eceab9530f8
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:36:13.418841 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.407364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55673 (local address 127.0.98.62:36445)
0619 16:36:13.407513 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:13.407517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:13.408302 (+ 785us) server_negotiation.cc:407] Connection header received
0619 16:36:13.409760 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:13.409763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:13.409821 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:13.409933 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:13.411011 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.411543 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.413022 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.413368 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.413498 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.415878 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:13.415881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:13.415883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:13.415906 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:13.417108 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:13.417654 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:13.417657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:13.417658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:13.417732 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:13.418024 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:13.418026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:13.418027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:13.418179 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:13.418252 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:13.418649 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 16:36:13.418733 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:36:13.419605 9040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.407706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:55673)
0619 16:36:13.408152 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:13.408191 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:13.409542 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:13.410096 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:13.410102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:13.410435 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:13.410869 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.410882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.411668 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.411678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.412895 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.412904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.413519 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.413523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.413723 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.414277 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:13.414291 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:13.415740 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:13.417209 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:13.417243 (+ 34us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:13.417252 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:13.417569 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:13.417812 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:13.417815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:13.417816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:13.417948 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:13.418261 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:13.418266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:13.418465 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 16:36:13.418607 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 16:36:13.418825 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":105,"threads_started":1}
I20260619 16:36:13.420599 9038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36445
I20260619 16:36:13.420830 9038 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:13.421269 9038 heartbeater.cc:511] Master 127.0.98.62:36445 requested a full tablet report, sending...
I20260619 16:36:13.422303 8579 ts_manager.cc:194] Registered new tserver with Master: c4f254371db14ad9835ce8d3d7035f30 (127.0.98.3:43337)
I20260619 16:36:13.422845 8579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55673
I20260619 16:36:13.426240 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 16:36:13.500775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 8641
I20260619 16:36:13.506275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 8775
I20260619 16:36:13.511439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 8909
I20260619 16:36:13.516364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 8547
I20260619 16:36:13.522414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36445
--webserver_interface=127.0.98.62
--webserver_port=34255
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:13.629617 9047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:13.629858 9047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:13.629899 9047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:13.633816 9047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:13.633865 9047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:13.633884 9047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:13.633900 9047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:13.633921 9047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:13.638897 9047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:13.639786 9047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:13.640688 9047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:13.648759 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:13.648849 9047 server_base.cc:1034] running on GCE node
W20260619 16:36:13.648778 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:13.648769 9053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:13.649574 9047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:13.650606 9047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:13.651845 9047 hybrid_clock.cc:648] HybridClock initialized: now 1781886973651830 us; error 26 us; skew 500 ppm
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:13.654645 9047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:36:13.655522 9047 webserver.cc:460] Webserver started at http://127.0.98.62:34255/ using document root <none> and password file <none>
I20260619 16:36:13.656071 9047 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:13.656175 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:13.659941 9047 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:13.661988 9062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.663189 9047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:13.663326 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "f1808f95ae4e445cb7a240b8d9914608"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "4f766c169251cfd30d86e4e30381e0ce"
server_key_iv: "9b39aecb12b1777418bdaaf055583510"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.663858 9047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:13.676478 9047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:13.679320 9047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:13.679528 9047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:13.686368 9047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36445
I20260619 16:36:13.686389 9114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36445 every 8 connection(s)
I20260619 16:36:13.687356 9047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:13.687855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9047
I20260619 16:36:13.688273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41549
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36445
--webserver_port=33773
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:36:13.691919 9115 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:36:13.693696 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Bootstrap starting.
I20260619 16:36:13.711228 9115 log.cc:824] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:13.717957 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:36:13.718325 9115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Bootstrap complete.
I20260619 16:36:13.722779 9115 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:13.723217 9115 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1808f95ae4e445cb7a240b8d9914608, State: Initialized, Role: FOLLOWER
I20260619 16:36:13.723868 9115 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:13.724087 9115 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:13.724211 9115 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:13.724367 9115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:36:13.726596 9115 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:13.726948 9115 leader_election.cc:302] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1808f95ae4e445cb7a240b8d9914608; no voters:
I20260619 16:36:13.727290 9115 leader_election.cc:288] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:36:13.727418 9119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:36:13.727648 9119 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [term 2 LEADER]: Becoming Leader. State: Replica: f1808f95ae4e445cb7a240b8d9914608, State: Running, Role: LEADER
I20260619 16:36:13.728018 9119 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } }
I20260619 16:36:13.728629 9115 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:13.729382 9120 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1808f95ae4e445cb7a240b8d9914608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } } }
I20260619 16:36:13.729496 9120 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:13.730423 9121 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1808f95ae4e445cb7a240b8d9914608. Latest consensus state: current_term: 2 leader_uuid: "f1808f95ae4e445cb7a240b8d9914608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1808f95ae4e445cb7a240b8d9914608" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36445 } } }
I20260619 16:36:13.730540 9121 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:13.730953 9126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:13.734411 9126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:13.734997 9126 catalog_manager.cc:1259] Loaded cluster ID: 81f01b60498c4b9c9e3695334175b352
I20260619 16:36:13.735052 9126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:13.737244 9126 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:13.737861 9126 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1808f95ae4e445cb7a240b8d9914608: Loaded TSK: 0
I20260619 16:36:13.739024 9126 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:36:13.799952 9117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:13.800197 9117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:13.800273 9117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:13.804710 9117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:13.804761 9117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:13.804870 9117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:13.810479 9117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33773
--tserver_master_addrs=127.0.98.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:13.811461 9117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:13.812466 9117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:13.822703 9145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:13.822670 9143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:13.822681 9142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:13.823031 9117 server_base.cc:1034] running on GCE node
I20260619 16:36:13.823454 9117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:13.824101 9117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:13.825274 9117 hybrid_clock.cc:648] HybridClock initialized: now 1781886973825254 us; error 41 us; skew 500 ppm
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886973, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:13.828063 9117 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 16:36:13.828960 9117 webserver.cc:460] Webserver started at http://127.0.98.1:33773/ using document root <none> and password file <none>
I20260619 16:36:13.829639 9117 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:13.829730 9117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:13.833635 9117 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260619 16:36:13.835644 9152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.836869 9117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:13.836956 9117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "32bd78d2576a4e0ea3999ee8aa8a19ab"
format_stamp: "Formatted at 2026-06-19 16:36:12 on dist-test-slave-0f28"
server_key: "c8271d431a766fd9fe36a50a3e9c8938"
server_key_iv: "68a272441adf3d0e304808be84de9858"
server_key_version: "encryptionkey@0"
I20260619 16:36:13.837553 9117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:13.851209 9117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:13.854321 9117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:13.854513 9117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:13.855125 9117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:13.855964 9117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:13.856010 9117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.856089 9117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:13.856122 9117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:13.866750 9117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41549
I20260619 16:36:13.866771 9265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41549 every 8 connection(s)
I20260619 16:36:13.867799 9117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:13.874202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9117
I20260619 16:36:13.874694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45841
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36445
--webserver_port=35787
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:36:13.882200 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:13.869938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50917 (local address 127.0.98.62:36445)
0619 16:36:13.870395 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:13.870414 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:13.870738 (+ 324us) server_negotiation.cc:407] Connection header received
0619 16:36:13.872494 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:13.872505 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:13.872778 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:13.873041 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:13.874455 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.875957 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.877635 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.878077 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.878257 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.881499 (+ 3242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50917: BlockingRecv error: recv got EOF from 127.0.98.1:50917 (error 108)
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":107,"threads_started":1}
I20260619 16:36:13.882210 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.870053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:50917)
0619 16:36:13.870561 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:13.870616 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:13.872133 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:13.873176 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:13.873184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:13.873665 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:13.874293 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.874312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.876133 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.876138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.877513 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.877522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.878240 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.878245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.878422 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.879043 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:13.879059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:13.881282 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":142,"threads_started":1}
W20260619 16:36:13.882762 9266 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36445 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36445: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:13.888240 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.883226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:55887)
0619 16:36:13.883333 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:13.883349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:13.883421 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:13.883663 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:13.883666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:13.883816 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:13.883895 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.883900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.884614 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.884619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.885141 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.885149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.885654 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.885658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.885753 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.886234 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:13.886244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:13.888095 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:36:13.888324 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:13.883313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55887 (local address 127.0.98.62:36445)
0619 16:36:13.883426 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:13.883430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:13.883444 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:13.883518 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:13.883520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:13.883566 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:13.883650 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:13.884004 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.884495 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.885258 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.885546 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.885652 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.888209 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55887: BlockingRecv error: recv got EOF from 127.0.98.1:55887 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:13 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:13.894308 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:13.888707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:35479)
0619 16:36:13.889042 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:13.889056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:13.889148 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:13.889406 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:13.889409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:13.889614 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:13.889716 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.889722 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.890380 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.890386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.891382 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:13.891392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.891805 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.891808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:13.891909 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.892337 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:13.892351 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:13.894150 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":96,"threads_started":1}
W20260619 16:36:13.894328 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:13.888828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35479 (local address 127.0.98.62:36445)
0619 16:36:13.888929 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:13.888935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:13.889132 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:36:13.889239 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:13.889241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:13.889278 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:13.889396 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:13.889818 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.890273 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.891487 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:13.891697 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:13.891768 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:13.894254 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35479: BlockingRecv error: recv got EOF from 127.0.98.1:35479 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:36:13.894552 9266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:36:13.984266 9270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:13.984534 9270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:13.984611 9270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:13.988610 9270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:13.988698 9270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:13.988794 9270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:13.994055 9270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35787
--tserver_master_addrs=127.0.98.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:13.995024 9270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:13.995986 9270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:14.005358 9276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:14.005388 9279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:14.005334 9277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:14.005568 9270 server_base.cc:1034] running on GCE node
I20260619 16:36:14.005913 9270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:14.006539 9270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:14.007721 9270 hybrid_clock.cc:648] HybridClock initialized: now 1781886974007707 us; error 26 us; skew 500 ppm
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:14.010491 9270 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 16:36:14.011364 9270 webserver.cc:460] Webserver started at http://127.0.98.2:35787/ using document root <none> and password file <none>
I20260619 16:36:14.011898 9270 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:14.012002 9270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:14.015654 9270 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:14.017647 9286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:14.018849 9270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:36:14.018981 9270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "d79b9879be2a4b84b5cc610fea69e67b"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "9d28f6bf1500c0e0d3967b1554697cdf"
server_key_iv: "4c59911eb21c40c40f25f40fd610771d"
server_key_version: "encryptionkey@0"
I20260619 16:36:14.019513 9270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:14.035576 9270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:14.038475 9270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:14.038691 9270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:14.039280 9270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:14.040161 9270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:14.040231 9270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:14.040294 9270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:14.040345 9270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:14.049165 9270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45841
I20260619 16:36:14.049214 9399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45841 every 8 connection(s)
I20260619 16:36:14.050168 9270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:14.051052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9270
I20260619 16:36:14.051509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43337
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36445
--webserver_port=35961
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:36:14.061971 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.052179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50973 (local address 127.0.98.62:36445)
0619 16:36:14.052345 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.052353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.053193 (+ 840us) server_negotiation.cc:407] Connection header received
0619 16:36:14.054652 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.054655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.054719 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.054801 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.056045 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.056576 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.057831 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.058067 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.058181 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.061869 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50973: BlockingRecv error: recv got EOF from 127.0.98.2:50973 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 16:36:14.062678 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.052525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:50973)
0619 16:36:14.053050 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.053091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.054420 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.055011 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.055022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.055397 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.055877 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.055890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.056721 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.056728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.057710 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.057720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.058225 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.058229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.058390 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.059157 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.059173 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.061749 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":159,"threads_started":1}
W20260619 16:36:14.063102 9400 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36445 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36445: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:14.069105 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.063523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:35105)
0619 16:36:14.063816 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.063829 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.063909 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.064141 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.064144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.064304 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.064407 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.064413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.065089 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.065092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.066023 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.066033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.066469 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.066473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.066564 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.067029 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.067043 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.068975 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":120}
W20260619 16:36:14.069226 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.063613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35105 (local address 127.0.98.62:36445)
0619 16:36:14.063740 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.063744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.063890 (+ 146us) server_negotiation.cc:407] Connection header received
0619 16:36:14.063996 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.063998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.064038 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.064104 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.064514 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.064976 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.066155 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.066369 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.066448 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.069097 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35105: BlockingRecv error: recv got EOF from 127.0.98.2:35105 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:14.074635 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.069546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:41359)
0619 16:36:14.069868 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.069880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.069987 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.070208 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.070210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.070393 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.070475 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.070479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.071164 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.071169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.071742 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.071750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.072187 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.072190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.072295 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.072781 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.072793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.074464 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":97,"threads_started":1}
W20260619 16:36:14.074662 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.069635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41359 (local address 127.0.98.62:36445)
0619 16:36:14.069738 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.069744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.069949 (+ 205us) server_negotiation.cc:407] Connection header received
0619 16:36:14.070068 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.070071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.070113 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.070194 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.070582 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.071061 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.071856 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.072069 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.072179 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.074567 (+ 2388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41359: BlockingRecv error: recv got EOF from 127.0.98.2:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 16:36:14.074947 9400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:36:14.160385 9402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:14.160617 9402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:14.160657 9402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:14.164579 9402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:14.164628 9402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:14.164722 9402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:14.169840 9402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35961
--tserver_master_addrs=127.0.98.62:36445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:14.170774 9402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:14.171720 9402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:14.180843 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:14.180825 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:14.181018 9402 server_base.cc:1034] running on GCE node
W20260619 16:36:14.180827 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:14.181533 9402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:14.182091 9402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:14.183276 9402 hybrid_clock.cc:648] HybridClock initialized: now 1781886974183244 us; error 46 us; skew 500 ppm
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886974, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:14.185880 9402 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 16:36:14.186717 9402 webserver.cc:460] Webserver started at http://127.0.98.3:35961/ using document root <none> and password file <none>
I20260619 16:36:14.187268 9402 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:14.187378 9402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:14.190955 9402 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:14.192894 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:14.194161 9402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:14.194293 9402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "c4f254371db14ad9835ce8d3d7035f30"
format_stamp: "Formatted at 2026-06-19 16:36:13 on dist-test-slave-0f28"
server_key: "e6dd3f63858ea9a4972db4e481bf1ad2"
server_key_iv: "14fb8535ae6a62eae0ee92059430598b"
server_key_version: "encryptionkey@0"
I20260619 16:36:14.194808 9402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:14.214546 9402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:14.217195 9402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:14.217442 9402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:14.218008 9402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:14.218887 9402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:14.218963 9402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:14.219026 9402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:14.219076 9402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:14.228689 9402 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43337
I20260619 16:36:14.228709 9532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43337 every 8 connection(s)
I20260619 16:36:14.229843 9402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:14.237716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9402
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 16:36:14.242419 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.231989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53353 (local address 127.0.98.62:36445)
0619 16:36:14.232122 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.232126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.233005 (+ 879us) server_negotiation.cc:407] Connection header received
0619 16:36:14.234557 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.234561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.234639 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.234748 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.235996 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.236576 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.237928 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.238266 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.238388 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.242294 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53353: BlockingRecv error: recv got EOF from 127.0.98.3:53353 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:36:14.243117 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.232344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:53353)
0619 16:36:14.232811 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.232873 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.234324 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.234936 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.234944 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.235327 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.235824 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.235838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.236714 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.236718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.237757 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.237767 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.238433 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.238438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.238639 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.239279 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.239303 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.242196 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":107,"threads_started":1}
W20260619 16:36:14.243709 9533 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36445 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36445: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:14.249225 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.238787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.0.98.62:36445)
0619 16:36:14.239067 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.239073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.239089 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:36:14.239125 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.239128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.239176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.239271 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.240276 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.240897 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.242268 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.242530 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.242620 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.244562 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:14.244570 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:14.244588 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:14.244620 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:14.246738 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:14.247257 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:14.247262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:14.247266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:14.247437 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:14.247817 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:14.247820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:14.247824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:14.248287 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:14.248495 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:14.248776 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:36:14.249021 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:14.250784 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.244184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:41915)
0619 16:36:14.244304 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.244319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.244398 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.244594 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.244597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.244767 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.244893 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.244902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.245898 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.245902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.246775 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.246787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.247344 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.247347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.247446 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.247935 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.247949 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.250678 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 16:36:14.250924 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.244291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41915 (local address 127.0.98.62:36445)
0619 16:36:14.244403 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.244406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.244417 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:36:14.244462 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.244464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.244496 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.244555 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.245021 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.245785 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.246886 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.247235 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.247327 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.250789 (+ 3462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41915: BlockingRecv error: recv got EOF from 127.0.98.3:41915 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:14.256215 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.251435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:45797)
0619 16:36:14.251534 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.251546 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.251603 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.252057 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.252059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.252178 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.252293 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.252300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.252973 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.252975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.253571 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.253579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.254044 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.254047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.254101 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.254530 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.254548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.256133 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 16:36:14.256395 9268 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.251692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45797 (local address 127.0.98.62:36445)
0619 16:36:14.251824 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.251827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.251838 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:36:14.251864 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.251866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.251918 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.252027 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.252427 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.252886 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.253691 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.253906 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.253990 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.256261 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45797: BlockingRecv error: recv got EOF from 127.0.98.3:45797 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 16:36:14.256457 9533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:14 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:14.901240 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:14.895174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:37421)
0619 16:36:14.895453 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:14.895472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:14.895580 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:14.895797 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:14.895799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:14.895984 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:14.896076 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.896081 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.897006 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.897010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.898124 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:14.898134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.898554 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.898557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:14.898634 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.899201 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:14.899218 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:14.901117 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":69,"threads_started":1}
W20260619 16:36:14.901397 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:14.895309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37421 (local address 127.0.98.62:36445)
0619 16:36:14.895576 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:14.895580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:14.895592 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:36:14.895645 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:14.895648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:14.895686 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:14.895759 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:14.896181 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.896863 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.898227 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:14.898443 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:14.898522 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:14.901246 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37421: BlockingRecv error: recv got EOF from 127.0.98.1:37421 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
Jun 19 16:36:15 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:15 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:15.081074 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:15.075602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:35841)
0619 16:36:15.075867 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:15.075881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:15.075984 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:15.076246 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:15.076249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:15.076467 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:15.076581 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:15.076587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.077377 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.077415 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:15.078011 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:15.078019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.078503 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.078507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:15.078584 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:15.079064 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:15.079075 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:15.080941 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
W20260619 16:36:15.081215 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:15.075697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35841 (local address 127.0.98.62:36445)
0619 16:36:15.075879 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:15.075883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:15.075933 (+ 50us) server_negotiation.cc:407] Connection header received
0619 16:36:15.076078 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:15.076080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:15.076122 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:15.076193 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:15.076700 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.077241 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.078139 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.078348 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.078455 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:15.081093 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35841: BlockingRecv error: recv got EOF from 127.0.98.2:35841 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 16:36:15 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:15 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:15.262720 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:15.257069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:50371)
0619 16:36:15.257390 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:15.257404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:15.257503 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:15.257738 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:15.257741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:15.257908 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:15.258012 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:15.258017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.258829 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.258832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:15.259383 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:15.259391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.259803 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.259807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:15.259937 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:15.260516 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:15.260544 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:15.262600 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":68,"threads_started":1}
W20260619 16:36:15.262917 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:15.257220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50371 (local address 127.0.98.62:36445)
0619 16:36:15.257394 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:15.257399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:15.257468 (+ 69us) server_negotiation.cc:407] Connection header received
0619 16:36:15.257576 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:15.257578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:15.257631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:15.257699 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:15.258161 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.258720 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.259489 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.259696 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.259774 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:15.262782 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50371: BlockingRecv error: recv got EOF from 127.0.98.3:50371 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 16:36:15 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:15 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:15.907660 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:15.902026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:56773)
0619 16:36:15.902315 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:15.902330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:15.902424 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:15.902666 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:15.902668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:15.902841 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:15.902954 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:15.902959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.903843 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.903846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:15.904426 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:15.904434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.904968 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.904976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:15.905052 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:15.905714 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:15.905726 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:15.907528 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
W20260619 16:36:15.907888 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:15.902092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56773 (local address 127.0.98.62:36445)
0619 16:36:15.902329 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:15.902333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:15.902379 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:36:15.902502 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:15.902504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:15.902550 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:15.902630 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:15.903084 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.903736 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.904544 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:15.904783 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:15.904875 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:15.907691 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56773: BlockingRecv error: recv got EOF from 127.0.98.1:56773 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":60,"threads_started":1}
Jun 19 16:36:16 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:16 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:16.088063 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:16.082027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:47049)
0619 16:36:16.082320 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:16.082334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:16.082432 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:16.082711 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:16.082713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:16.082883 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:16.082973 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:16.082978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.083730 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.083733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:16.084461 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:16.084474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.085074 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.085079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:16.085177 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:16.085871 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:16.085889 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:16.087930 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
W20260619 16:36:16.088228 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:16.082095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47049 (local address 127.0.98.62:36445)
0619 16:36:16.082266 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:16.082269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:16.082410 (+ 141us) server_negotiation.cc:407] Connection header received
0619 16:36:16.082543 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:16.082546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:16.082592 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:16.082677 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:16.083075 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.083604 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.084620 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.084933 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.085044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:16.088066 (+ 3022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47049: BlockingRecv error: recv got EOF from 127.0.98.2:47049 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 16:36:16 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:16 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:16.268960 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:16.263489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:49007)
0619 16:36:16.263782 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:16.263797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:16.263887 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:16.264157 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:16.264159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:16.264336 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:16.264431 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:16.264436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.265202 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.265205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:16.265792 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:16.265800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.266259 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.266262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:16.266325 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:16.266996 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:16.267007 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:16.268832 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":72,"threads_started":1}
W20260619 16:36:16.269167 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:16.263740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49007 (local address 127.0.98.62:36445)
0619 16:36:16.263863 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:16.263868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:16.263889 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:36:16.263972 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:16.263974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:16.264017 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:16.264083 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:16.264557 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.265093 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.265904 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.266148 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.266236 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:16.269034 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49007: BlockingRecv error: recv got EOF from 127.0.98.3:49007 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 16:36:16 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:16 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:16.913625 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:16.908390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:36355)
0619 16:36:16.908681 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:16.908695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:16.908782 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:16.909019 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:16.909020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:16.909198 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:16.909290 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:16.909295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.910052 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.910054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:16.910619 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:16.910626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.911084 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.911086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:16.911145 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:16.911683 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:16.911694 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:16.913468 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
W20260619 16:36:16.913697 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:16.908487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36355 (local address 127.0.98.62:36445)
0619 16:36:16.908728 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:16.908731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:16.908744 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:16.908866 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:16.908868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:16.908910 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:16.909003 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:16.909422 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.909946 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.910752 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:16.910967 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:16.911061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:16.913598 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36355: BlockingRecv error: recv got EOF from 127.0.98.1:36355 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
Jun 19 16:36:17 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:17 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:17.094837 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:17.088805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:59447)
0619 16:36:17.089139 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:17.089152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:17.089249 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:17.089561 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:17.089563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:17.089735 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:17.089827 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:17.089833 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.090748 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.090753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:17.091766 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:17.091774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.092306 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.092309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:17.092369 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:17.092884 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:17.092895 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:17.094710 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
W20260619 16:36:17.095049 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:17.088945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59447 (local address 127.0.98.62:36445)
0619 16:36:17.089098 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:17.089101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:17.089222 (+ 121us) server_negotiation.cc:407] Connection header received
0619 16:36:17.089382 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:17.089384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:17.089435 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:17.089511 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:17.089955 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.090588 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.091949 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.092204 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.092290 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:17.094837 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59447: BlockingRecv error: recv got EOF from 127.0.98.2:59447 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:36:17 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:17 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:17.275642 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:17.269744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:53013)
0619 16:36:17.270077 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:17.270093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:17.270242 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:17.270534 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:17.270538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:17.270773 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:17.270864 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:17.270870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.271700 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.271703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:17.272329 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:17.272337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.272873 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.272877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:17.272971 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:17.273543 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:17.273558 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:17.275441 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
W20260619 16:36:17.275697 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:17.269872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53013 (local address 127.0.98.62:36445)
0619 16:36:17.270037 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:17.270043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:17.270191 (+ 148us) server_negotiation.cc:407] Connection header received
0619 16:36:17.270356 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:17.270361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:17.270421 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:17.270514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:17.271001 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.271573 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.272460 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.272749 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.272873 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:17.275586 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53013: BlockingRecv error: recv got EOF from 127.0.98.3:53013 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 16:36:17 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:17 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:17.920132 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:17.914568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:42105)
0619 16:36:17.914934 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:17.914951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:17.915080 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:17.915392 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:17.915394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:17.915648 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:17.915789 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:17.915799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.916578 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.916580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:17.917147 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:17.917154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.917657 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.917660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:17.917721 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:17.918181 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:17.918192 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:17.920020 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":93,"threads_started":1}
W20260619 16:36:17.920344 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:17.914677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42105 (local address 127.0.98.62:36445)
0619 16:36:17.914997 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:17.915000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:17.915012 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:36:17.915183 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:17.915186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:17.915267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:17.915364 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:17.915914 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.916478 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.917247 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:17.917553 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:17.917657 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:17.920186 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42105: BlockingRecv error: recv got EOF from 127.0.98.1:42105 (error 108)
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":61,"threads_started":1}
Jun 19 16:36:18 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:18 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:18.101629 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:18.095622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:44877)
0619 16:36:18.095979 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:18.095993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:18.096135 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:18.096412 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:18.096414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:18.096622 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:18.096713 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:18.096718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.097596 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.097603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:18.098241 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:18.098251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.098778 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.098782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:18.098872 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:18.099436 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:18.099448 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:18.101515 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":87,"threads_started":1}
W20260619 16:36:18.101765 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:18.095750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44877 (local address 127.0.98.62:36445)
0619 16:36:18.095924 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:18.095929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:18.096073 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:36:18.096235 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:18.096239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:18.096297 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:18.096387 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:18.096832 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.097486 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.098364 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.098656 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.098764 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:18.101650 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44877: BlockingRecv error: recv got EOF from 127.0.98.2:44877 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 16:36:18 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:18 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:18.281740 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:18.276407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:58425)
0619 16:36:18.276677 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:18.276697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:18.276825 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:18.277093 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:18.277096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:18.277298 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:18.277409 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:18.277414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.278174 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.278179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:18.278728 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:18.278735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.279197 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.279201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:18.279285 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:18.279758 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:18.279769 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:18.281619 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
W20260619 16:36:18.281917 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:18.276458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58425 (local address 127.0.98.62:36445)
0619 16:36:18.276645 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:18.276650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:18.276774 (+ 124us) server_negotiation.cc:407] Connection header received
0619 16:36:18.276921 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:18.276924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:18.276975 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:18.277071 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:18.277531 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.278060 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.278842 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.279082 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.279196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:18.281755 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58425: BlockingRecv error: recv got EOF from 127.0.98.3:58425 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 16:36:18 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:18 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:18.926376 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:18.920939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:48487)
0619 16:36:18.921276 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:18.921290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:18.921420 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:18.921667 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:18.921669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:18.921842 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:18.921933 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:18.921938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.922727 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.922730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:18.923379 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:18.923389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.923847 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.923850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:18.923912 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:18.924432 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:18.924443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:18.926242 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
W20260619 16:36:18.926543 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:18.921053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48487 (local address 127.0.98.62:36445)
0619 16:36:18.921278 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:18.921282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:18.921392 (+ 110us) server_negotiation.cc:407] Connection header received
0619 16:36:18.921497 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:18.921499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:18.921542 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:18.921635 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:18.922060 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.922621 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.923521 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:18.923742 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:18.923831 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:18.926400 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48487: BlockingRecv error: recv got EOF from 127.0.98.1:48487 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":58,"threads_started":1}
Jun 19 16:36:19 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:19 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:19.108273 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:19.102448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:35897)
0619 16:36:19.102758 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:19.102774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:19.102889 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:19.103162 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:19.103165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:19.103374 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:19.103490 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:19.103496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.104245 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.104251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:19.104813 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:19.104821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.105288 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.105291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:19.105418 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:19.106064 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:19.106078 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:19.108105 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
W20260619 16:36:19.108318 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:19.102601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35897 (local address 127.0.98.62:36445)
0619 16:36:19.102759 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:19.102765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:19.102834 (+ 69us) server_negotiation.cc:407] Connection header received
0619 16:36:19.103005 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:19.103008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:19.103063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:19.103145 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:19.103609 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.104138 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.104939 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.105178 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.105286 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:19.108223 (+ 2937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35897: BlockingRecv error: recv got EOF from 127.0.98.2:35897 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 16:36:19 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:19 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:19.288179 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:19.282517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:54697)
0619 16:36:19.282809 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:19.282822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:19.282930 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:19.283273 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:19.283276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:19.283481 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:19.283572 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:19.283578 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.284615 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.284618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:19.285201 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:19.285209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.285627 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.285630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:19.285694 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:19.286254 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:19.286266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:19.288059 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":126,"threads_started":1}
W20260619 16:36:19.288300 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:19.282598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54697 (local address 127.0.98.62:36445)
0619 16:36:19.282736 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:19.282739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:19.282888 (+ 149us) server_negotiation.cc:407] Connection header received
0619 16:36:19.283072 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:19.283075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:19.283130 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:19.283212 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:19.283707 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.284499 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.285305 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.285526 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.285602 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:19.288183 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54697: BlockingRecv error: recv got EOF from 127.0.98.3:54697 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 16:36:19 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:19 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:19.933246 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:19.927239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:52975)
0619 16:36:19.927540 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:19.927554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:19.927641 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:19.927940 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:19.927943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:19.928123 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:19.928218 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:19.928224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.929028 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.929031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:19.929800 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:19.929815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.930333 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.930339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:19.930430 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:19.931091 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:19.931109 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:19.933066 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":82,"threads_started":1}
W20260619 16:36:19.933303 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:19.927360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52975 (local address 127.0.98.62:36445)
0619 16:36:19.927612 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:19.927615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:19.927628 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:19.927732 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:19.927735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:19.927777 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:19.927852 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:19.928324 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.928923 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.929941 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:19.930239 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:19.930319 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:19.933184 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52975: BlockingRecv error: recv got EOF from 127.0.98.1:52975 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
Jun 19 16:36:20 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:20 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:20.114892 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:20.109118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:50019)
0619 16:36:20.109491 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:20.109505 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:20.109603 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:20.109810 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:20.109812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:20.109980 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:20.110068 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:20.110073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.110917 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.110921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:20.111479 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:20.111487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.111937 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.111941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:20.112023 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:20.112610 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:20.112628 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:20.114780 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":96,"threads_started":1}
W20260619 16:36:20.115069 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:20.109187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50019 (local address 127.0.98.62:36445)
0619 16:36:20.109355 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:20.109401 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:20.109574 (+ 173us) server_negotiation.cc:407] Connection header received
0619 16:36:20.109668 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:20.109670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:20.109713 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:20.109777 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:20.110158 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.110743 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.111595 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.111819 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.111933 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:20.114916 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50019: BlockingRecv error: recv got EOF from 127.0.98.2:50019 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 16:36:20 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:20 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:20.294704 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:20.288984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:60897)
0619 16:36:20.289359 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:20.289372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:20.289503 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:20.289790 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:20.289793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:20.290004 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:20.290103 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:20.290109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.290905 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.290911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:20.291489 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:20.291496 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.291963 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.291966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:20.292050 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:20.292639 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:20.292651 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:20.294515 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":107,"threads_started":1}
W20260619 16:36:20.294737 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:20.289117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60897 (local address 127.0.98.62:36445)
0619 16:36:20.289286 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:20.289291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:20.289451 (+ 160us) server_negotiation.cc:407] Connection header received
0619 16:36:20.289622 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:20.289626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:20.289676 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:20.289763 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:20.290225 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.290790 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.291634 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.291845 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.291958 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:20.294638 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60897: BlockingRecv error: recv got EOF from 127.0.98.3:60897 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 16:36:20 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:20 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:20.939580 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:20.934065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:36525)
0619 16:36:20.934417 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:20.934431 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:20.934525 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:20.934759 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:20.934761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:20.934949 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:20.935054 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:20.935059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.935825 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.935828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:20.936404 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:20.936411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.936866 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.936869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:20.936982 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:20.937606 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:20.937618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:20.939467 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
W20260619 16:36:20.939762 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:20.934180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36525 (local address 127.0.98.62:36445)
0619 16:36:20.934435 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:20.934438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:20.934502 (+ 64us) server_negotiation.cc:407] Connection header received
0619 16:36:20.934605 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:20.934608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:20.934651 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:20.934726 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:20.935188 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.935721 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.936529 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:20.936738 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:20.936862 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:20.939593 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36525: BlockingRecv error: recv got EOF from 127.0.98.1:36525 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":64,"threads_started":1}
Jun 19 16:36:21 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:21 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:21.121500 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:21.115712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:41963)
0619 16:36:21.116021 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:21.116041 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:21.116142 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:21.116355 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:21.116357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:21.116525 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:21.116618 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:21.116623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.117513 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.117516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:21.118088 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:21.118097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.118533 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.118537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:21.118606 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:21.119089 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:21.119101 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:21.121318 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
W20260619 16:36:21.121608 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:21.115778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41963 (local address 127.0.98.62:36445)
0619 16:36:21.115978 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:21.115982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:21.116158 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:36:21.116203 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:21.116205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:21.116251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:21.116322 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:21.116750 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.117408 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.118210 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.118429 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.118515 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:21.121498 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41963: BlockingRecv error: recv got EOF from 127.0.98.2:41963 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 16:36:21 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:21 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:21.301090 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:21.295498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:55207)
0619 16:36:21.295852 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:21.295865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:21.295998 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:21.296268 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:21.296271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:21.296475 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:21.296562 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:21.296567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.297311 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.297343 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:21.297996 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:21.298004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.298443 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.298446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:21.298528 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:21.299151 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:21.299162 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:21.300914 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":82,"threads_started":1}
W20260619 16:36:21.301132 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:21.295598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55207 (local address 127.0.98.62:36445)
0619 16:36:21.295766 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:21.295771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:21.295939 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:36:21.296099 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:21.296104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:21.296159 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:21.296247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:21.296674 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.297201 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.298111 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.298333 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.298441 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:21.301035 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55207: BlockingRecv error: recv got EOF from 127.0.98.3:55207 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 16:36:21 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:21 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:21.946376 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:21.940376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:49747)
0619 16:36:21.940719 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:21.940734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:21.940900 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:21.941155 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:21.941157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:21.941370 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:21.941469 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:21.941475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.942245 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.942248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:21.942899 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:21.942909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.943336 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.943339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:21.943417 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:21.943996 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:21.944007 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:21.946255 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
W20260619 16:36:21.946581 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:21.940455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49747 (local address 127.0.98.62:36445)
0619 16:36:21.940759 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:21.940763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:21.940842 (+ 79us) server_negotiation.cc:407] Connection header received
0619 16:36:21.940986 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:21.940988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:21.941033 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:21.941116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:21.941573 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.942146 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.943004 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:21.943218 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:21.943292 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:21.946434 (+ 3142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49747: BlockingRecv error: recv got EOF from 127.0.98.1:49747 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
Jun 19 16:36:22 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:22 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:22.128057 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:22.122337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:59097)
0619 16:36:22.122641 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:22.122654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:22.122752 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:22.122999 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:22.123002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:22.123174 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:22.123266 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:22.123271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.124309 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.124312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:22.124887 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:22.124896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.125456 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.125461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:22.125538 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:22.126125 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:22.126137 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:22.127939 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
W20260619 16:36:22.128315 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:22.122392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59097 (local address 127.0.98.62:36445)
0619 16:36:22.122630 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:22.122634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:22.122740 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:36:22.122818 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:22.122820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:22.122865 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:22.122936 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:22.123525 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.124193 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.124998 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.125232 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.125424 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:22.128167 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59097: BlockingRecv error: recv got EOF from 127.0.98.2:59097 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Jun 19 16:36:22 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:22 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:22.307461 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:22.301954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:54985)
0619 16:36:22.302211 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:22.302224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:22.302318 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:22.302645 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:22.302648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:22.302849 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:22.302938 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:22.302943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.303683 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.303689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:22.304242 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:22.304249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.304699 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.304702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:22.304786 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:22.305363 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:22.305400 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:22.307288 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
W20260619 16:36:22.307518 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:22.302006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54985 (local address 127.0.98.62:36445)
0619 16:36:22.302142 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:22.302147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:22.302276 (+ 129us) server_negotiation.cc:407] Connection header received
0619 16:36:22.302469 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:22.302474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:22.302532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:22.302621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:22.303056 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.303575 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.304361 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.304585 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.304695 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:22.307412 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54985: BlockingRecv error: recv got EOF from 127.0.98.3:54985 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 16:36:22 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:22 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:22.955822 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:22.947101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:45517)
0619 16:36:22.947364 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:22.947377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:22.947459 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:22.947745 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:22.947747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:22.947920 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:22.948011 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:22.948016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.948884 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.948887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:22.949509 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:22.949518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.949990 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.949992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:22.950051 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:22.953573 (+ 3522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:22.953585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:22.955712 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
W20260619 16:36:22.956061 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:22.947240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45517 (local address 127.0.98.62:36445)
0619 16:36:22.947533 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:22.947537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:22.947550 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:22.947584 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:22.947586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:22.947628 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:22.947714 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:22.948159 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.948758 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.949627 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:22.949889 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:22.949976 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:22.955908 (+ 5932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45517: BlockingRecv error: recv got EOF from 127.0.98.1:45517 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
Jun 19 16:36:23 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:23 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:23.134837 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:23.128885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:34001)
0619 16:36:23.129172 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:23.129185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:23.129313 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:23.129640 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:23.129644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:23.129893 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:23.129985 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:23.129991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.130740 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.130746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:23.131314 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:23.131321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.131771 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.131774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:23.131852 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:23.132432 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:23.132443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:23.134669 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
W20260619 16:36:23.134908 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:23.128940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34001 (local address 127.0.98.62:36445)
0619 16:36:23.129109 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:23.129114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:23.129259 (+ 145us) server_negotiation.cc:407] Connection header received
0619 16:36:23.129437 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:23.129440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:23.129492 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:23.129618 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:23.130102 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.130631 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.131433 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.131660 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.131770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:23.134786 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34001: BlockingRecv error: recv got EOF from 127.0.98.2:34001 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:36:23 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:23 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:23.313855 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:23.308297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:58447)
0619 16:36:23.308592 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:23.308610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:23.308757 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:23.309082 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:23.309085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:23.309306 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:23.309518 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:23.309526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.310294 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.310297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:23.310887 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:23.310895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.311290 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.311293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:23.311349 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:23.311914 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:23.311928 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:23.313746 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
W20260619 16:36:23.314082 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:23.308425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58447 (local address 127.0.98.62:36445)
0619 16:36:23.308577 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:23.308582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:23.308698 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:36:23.308881 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:23.308884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:23.308944 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:23.309031 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:23.309638 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.310195 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.310988 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.311201 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.311272 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:23.313965 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58447: BlockingRecv error: recv got EOF from 127.0.98.3:58447 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 16:36:23 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:23 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:23.962536 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:23.956590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:53615)
0619 16:36:23.956873 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:23.956887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:23.956967 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:23.957238 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:23.957240 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:23.957453 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:23.957550 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:23.957556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.958322 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.958327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:23.959096 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:23.959127 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.959757 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.959760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:23.959823 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:23.960384 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:23.960397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:23.962358 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
W20260619 16:36:23.962610 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:23.956699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53615 (local address 127.0.98.62:36445)
0619 16:36:23.956956 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:23.956959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:23.956973 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:23.957053 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:23.957055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:23.957094 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:23.957169 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:23.957661 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.958190 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.959261 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:23.959638 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:23.959732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:23.962488 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53615: BlockingRecv error: recv got EOF from 127.0.98.1:53615 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
Jun 19 16:36:24 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:24 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:24.141559 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:24.135650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:45491)
0619 16:36:24.135957 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:24.135971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:24.136078 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:24.136326 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:24.136328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:24.136545 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:24.136639 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:24.136644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.137454 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.137457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:24.138017 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:24.138025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.138506 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.138509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:24.138573 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:24.139054 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:24.139065 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:24.141420 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":79,"threads_started":1}
W20260619 16:36:24.141719 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:24.135754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45491 (local address 127.0.98.62:36445)
0619 16:36:24.135918 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:24.135921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:24.136041 (+ 120us) server_negotiation.cc:407] Connection header received
0619 16:36:24.136163 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:24.136165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:24.136210 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:24.136286 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:24.136739 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.137301 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.138121 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.138388 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.138474 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:24.141576 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45491: BlockingRecv error: recv got EOF from 127.0.98.2:45491 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 16:36:24 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:24 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:24.319780 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:24.314674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:50937)
0619 16:36:24.314954 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:24.314967 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:24.315057 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:24.315269 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:24.315271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:24.315471 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:24.315563 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:24.315568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.316272 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.316274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:24.316837 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:24.316844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.317247 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.317250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:24.317310 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:24.317828 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:24.317839 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:24.319664 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":114,"threads_started":1}
W20260619 16:36:24.319882 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:24.314830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50937 (local address 127.0.98.62:36445)
0619 16:36:24.314968 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:24.314972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:24.315016 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:36:24.315120 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:24.315122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:24.315165 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:24.315230 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:24.315663 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.316181 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.316936 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.317153 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.317225 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:24.319794 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50937: BlockingRecv error: recv got EOF from 127.0.98.3:50937 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 16:36:24 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:24 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:24.969183 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:24.963368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:58241)
0619 16:36:24.963665 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:24.963680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:24.963772 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:24.964043 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:24.964046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:24.964243 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:24.964360 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:24.964366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.965161 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.965164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:24.965843 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:24.965853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.966301 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.966304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:24.966393 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:24.966982 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:24.966994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:24.968999 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
W20260619 16:36:24.969239 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:24.963481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58241 (local address 127.0.98.62:36445)
0619 16:36:24.963729 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:24.963733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:24.963746 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:36:24.963851 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:24.963853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:24.963904 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:24.963995 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:24.964491 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.965052 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.965949 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:24.966170 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:24.966255 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:24.969121 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58241: BlockingRecv error: recv got EOF from 127.0.98.1:58241 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
Jun 19 16:36:25 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:25 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:25.148087 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:25.142387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:57493)
0619 16:36:25.142650 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:25.142663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:25.142762 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:25.142988 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:25.142990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:25.143164 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:25.143270 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:25.143282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.144280 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.144284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:25.145105 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:25.145116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.145611 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.145614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:25.145688 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:25.146129 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:25.146140 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:25.147970 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":64,"threads_started":1}
W20260619 16:36:25.148214 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:25.142508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57493 (local address 127.0.98.62:36445)
0619 16:36:25.142676 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:25.142679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:25.142725 (+ 46us) server_negotiation.cc:407] Connection header received
0619 16:36:25.142838 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:25.142840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:25.142885 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:25.142953 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:25.143403 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.144135 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.145207 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.145510 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.145593 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:25.148095 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57493: BlockingRecv error: recv got EOF from 127.0.98.2:57493 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:36:25 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:25 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:25.326535 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:25.320804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:53019)
0619 16:36:25.321160 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:25.321180 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:25.321333 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:25.321603 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:25.321606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:25.321815 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:25.321904 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:25.321909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.322804 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.322811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:25.323432 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:25.323450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.323959 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.323962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:25.324050 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:25.324547 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:25.324558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:25.326415 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":144,"threads_started":1}
W20260619 16:36:25.326846 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:25.320963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53019 (local address 127.0.98.62:36445)
0619 16:36:25.321119 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:25.321123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:25.321256 (+ 133us) server_negotiation.cc:407] Connection header received
0619 16:36:25.321431 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:25.321436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:25.321496 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:25.321585 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:25.322018 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.322680 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.323562 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.323842 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.323962 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:25.326570 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53019: BlockingRecv error: recv got EOF from 127.0.98.3:53019 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 16:36:25 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:25 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:25.975257 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:25.969997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:45739)
0619 16:36:25.970282 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:25.970296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:25.970417 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:25.970674 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:25.970677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:25.970857 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:25.970950 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:25.970956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.971703 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.971706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:25.972256 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:25.972264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.972723 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.972726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:25.972792 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:25.973394 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:25.973406 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:25.975133 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
W20260619 16:36:25.975477 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:25.970063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45739 (local address 127.0.98.62:36445)
0619 16:36:25.970290 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:25.970294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:25.970412 (+ 118us) server_negotiation.cc:407] Connection header received
0619 16:36:25.970500 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:25.970502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:25.970548 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:25.970619 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:25.971062 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.971594 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.972387 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:25.972607 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:25.972686 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:25.975276 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45739: BlockingRecv error: recv got EOF from 127.0.98.1:45739 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
Jun 19 16:36:26 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:26 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:26.154639 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:26.149006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:46343)
0619 16:36:26.149322 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:26.149380 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:26.149473 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:26.149783 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:26.149786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:26.150000 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:26.150089 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:26.150094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.150842 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.150848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:26.151491 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:26.151501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.152040 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.152044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:26.152126 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:26.152669 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:26.152691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:26.154473 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":84,"threads_started":1}
W20260619 16:36:26.154714 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:26.149115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46343 (local address 127.0.98.62:36445)
0619 16:36:26.149278 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:26.149283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:26.149588 (+ 305us) server_negotiation.cc:407] Connection header received
0619 16:36:26.149637 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:26.149639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:26.149680 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:26.149764 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:26.150206 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.150732 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.151620 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.151911 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.152036 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:26.154606 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46343: BlockingRecv error: recv got EOF from 127.0.98.2:46343 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 16:36:26 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:26 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:26.332839 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:26.327307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:50457)
0619 16:36:26.327627 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:26.327640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:26.327765 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:26.328034 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:26.328037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:26.328290 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:26.328388 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:26.328393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.329171 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.329175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:26.329833 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:26.329842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.330379 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.330382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:26.330468 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:26.330979 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:26.330991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:26.332728 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":86,"threads_started":1}
W20260619 16:36:26.333009 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:26.327359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50457 (local address 127.0.98.62:36445)
0619 16:36:26.327513 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:26.327518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:26.327712 (+ 194us) server_negotiation.cc:407] Connection header received
0619 16:36:26.327872 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:26.327875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:26.327930 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:26.328014 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:26.328517 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.329048 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.329952 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.330268 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.330376 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:26.332856 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50457: BlockingRecv error: recv got EOF from 127.0.98.3:50457 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 16:36:26 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:26 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:26.982829 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:26.976054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:47209)
0619 16:36:26.976303 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:26.976318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:26.976416 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:26.976753 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:26.976758 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:26.977018 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:26.977182 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:26.977190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.978253 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.978259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:26.978877 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:26.978886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.979640 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.979644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:26.979722 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:26.980333 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:26.980347 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:26.982625 (+ 2278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
W20260619 16:36:26.982857 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:26.976187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47209 (local address 127.0.98.62:36445)
0619 16:36:26.976459 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:26.976465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:26.976480 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:36:26.976529 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:26.976533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:26.976589 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:26.976689 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:26.977351 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.978139 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.979002 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:26.979395 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:26.979498 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:26.982739 (+ 3241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47209: BlockingRecv error: recv got EOF from 127.0.98.1:47209 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
Jun 19 16:36:27 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:27 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:27.162154 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:27.155552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:43035)
0619 16:36:27.155841 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:27.155855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:27.155951 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:27.156246 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:27.156253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:27.156445 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:27.156542 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:27.156547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.157716 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.157720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:27.158606 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:27.158616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.159165 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.159168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:27.159248 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:27.159817 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:27.159830 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:27.161984 (+ 2154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":116,"threads_started":1}
W20260619 16:36:27.162209 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:27.155695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43035 (local address 127.0.98.62:36445)
0619 16:36:27.155872 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:27.155875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:27.155906 (+ 31us) server_negotiation.cc:407] Connection header received
0619 16:36:27.156042 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:27.156044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:27.156125 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:27.156193 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:27.156664 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.157605 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.158772 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.159041 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.159151 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:27.162104 (+ 2953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43035: BlockingRecv error: recv got EOF from 127.0.98.2:43035 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 16:36:27 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:27 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:27.338697 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:27.333557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:36669)
0619 16:36:27.333866 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:27.333878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:27.333979 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:27.334216 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:27.334218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:27.334391 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:27.334485 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:27.334490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.335183 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.335186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:27.335738 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:27.335745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.336212 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.336215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:27.336278 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:27.336749 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:27.336760 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:27.338585 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":81,"threads_started":1}
W20260619 16:36:27.338836 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:27.333662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36669 (local address 127.0.98.62:36445)
0619 16:36:27.333786 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:27.333789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:27.333943 (+ 154us) server_negotiation.cc:407] Connection header received
0619 16:36:27.334066 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:27.334068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:27.334112 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:27.334179 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:27.334586 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.335092 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.335848 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.336106 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.336187 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:27.338714 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36669: BlockingRecv error: recv got EOF from 127.0.98.3:36669 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 16:36:27 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:27 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:27.994138 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:27.983678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:58161)
0619 16:36:27.983946 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:27.983963 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:27.984071 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:27.984430 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:27.984434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:27.984693 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:27.984839 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:27.984847 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.985729 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.985731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:27.986277 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:27.986285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.986771 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.986773 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:27.986831 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:27.991854 (+ 5023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:27.991867 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:27.994025 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
W20260619 16:36:27.994328 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:27.983728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58161 (local address 127.0.98.62:36445)
0619 16:36:27.984074 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:27.984079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:27.984093 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:27.984186 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:27.984189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:27.984258 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:27.984353 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:27.984974 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.985630 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.986438 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:27.986662 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:27.986748 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:27.994153 (+ 7405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58161: BlockingRecv error: recv got EOF from 127.0.98.1:58161 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":95,"threads_started":1}
Jun 19 16:36:28 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:28 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:28.168910 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:28.162856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:52769)
0619 16:36:28.163191 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:28.163208 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:28.163354 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:28.163628 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:28.163631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:28.163833 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:28.163936 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:28.163945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.164685 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.164690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:28.165274 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:28.165281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.165807 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.165811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:28.165910 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:28.166469 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:28.166485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:28.168793 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":89,"threads_started":1}
W20260619 16:36:28.169183 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:28.163002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52769 (local address 127.0.98.62:36445)
0619 16:36:28.163172 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:28.163177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:28.163293 (+ 116us) server_negotiation.cc:407] Connection header received
0619 16:36:28.163464 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:28.163468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:28.163522 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:28.163608 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:28.164056 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.164577 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.165474 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.165694 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.165803 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:28.168982 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52769: BlockingRecv error: recv got EOF from 127.0.98.2:52769 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 16:36:28 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:28 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:28.344875 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:28.339500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.3:46141)
0619 16:36:28.339757 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:28.339776 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:28.339876 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:28.340151 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:28.340153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:28.340326 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:28.340429 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:28.340434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.341258 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.341261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:28.341891 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:28.341900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.342306 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.342309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:28.342411 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:28.342932 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:28.342944 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:28.344726 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
W20260619 16:36:28.344992 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:28.339609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46141 (local address 127.0.98.62:36445)
0619 16:36:28.339764 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:28.339768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:28.339844 (+ 76us) server_negotiation.cc:407] Connection header received
0619 16:36:28.339963 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:28.339965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:28.340009 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:28.340095 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:28.340551 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.341139 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.341998 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.342212 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.342292 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:28.344852 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46141: BlockingRecv error: recv got EOF from 127.0.98.3:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 16:36:28 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:28 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:29.000810 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:28.995048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.1:49845)
0619 16:36:28.995312 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:28.995325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:28.995444 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:28.995698 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:28.995700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:28.995879 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:28.995969 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:28.995974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.996773 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.996776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:28.997372 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:28.997380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.997836 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.997839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:28.997898 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:28.998531 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:28.998548 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:29.000701 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
W20260619 16:36:29.000963 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:28.995127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49845 (local address 127.0.98.62:36445)
0619 16:36:28.995373 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:28.995377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:28.995404 (+ 27us) server_negotiation.cc:407] Connection header received
0619 16:36:28.995530 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:28.995532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:28.995577 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:28.995657 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:28.996096 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.996653 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.997496 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:28.997735 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:28.997842 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.000835 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49845: BlockingRecv error: recv got EOF from 127.0.98.1:49845 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
Jun 19 16:36:29 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 16:36:29 dist-test-slave-0f28 krb5kdc[8534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 16:36:29.175494 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:29.169868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36445 (local address 127.0.98.2:36171)
0619 16:36:29.170212 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:29.170230 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:29.170332 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:29.170652 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:29.170655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:29.170887 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:29.170997 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:29.171003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.171754 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.171758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:29.172350 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:29.172357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.172804 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.172807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:29.172890 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.173473 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:29.173485 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:29.175268 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36445: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
W20260619 16:36:29.175496 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:29.169995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36171 (local address 127.0.98.62:36445)
0619 16:36:29.170228 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:29.170231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:29.170311 (+ 80us) server_negotiation.cc:407] Connection header received
0619 16:36:29.170447 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:29.170451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:29.170539 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:29.170631 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:29.171116 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.171648 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.172477 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.172696 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.172800 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.175396 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36171: BlockingRecv error: recv got EOF from 127.0.98.2:36171 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:36:29.238230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9117
I20260619 16:36:29.244338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9270
I20260619 16:36:29.252559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9402
I20260619 16:36:29.257882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9047
2026-06-19T16:36:29Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17876 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T16:36:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:29Z Disabled control of system clock
I20260619 16:36:29.287429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35197
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 16:36:29.397611 9600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:29.397846 9600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:29.397881 9600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:29.401769 9600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:29.401815 9600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:29.401831 9600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:29.401847 9600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:29.401862 9600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:29.406468 9600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:29.407356 9600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:29.408586 9600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:36:29.416710 9600 server_base.cc:1034] running on GCE node
W20260619 16:36:29.416626 9605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:29.416615 9606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:29.416616 9608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:29.417484 9600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:29.418452 9600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:29.419616 9600 hybrid_clock.cc:648] HybridClock initialized: now 1781886989419583 us; error 43 us; skew 500 ppm
I20260619 16:36:29.421425 9600 webserver.cc:460] Webserver started at http://127.0.98.62:46129/ using document root <none> and password file <none>
I20260619 16:36:29.422034 9600 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:29.422122 9600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:29.422324 9600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:29.424207 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "bc32a032e8f04be4a78ab542bb62b429"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "3c1ce2f3aa0f438be6e8bd7b59c33041"
server_key_iv: "69287ed86f2bb51c270ed0c77e3dbe6a"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.424677 9600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc32a032e8f04be4a78ab542bb62b429"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "3c1ce2f3aa0f438be6e8bd7b59c33041"
server_key_iv: "69287ed86f2bb51c270ed0c77e3dbe6a"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.427994 9600 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:36:29.430239 9614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.431537 9600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:29.431661 9600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "bc32a032e8f04be4a78ab542bb62b429"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "3c1ce2f3aa0f438be6e8bd7b59c33041"
server_key_iv: "69287ed86f2bb51c270ed0c77e3dbe6a"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.431762 9600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:29.442735 9600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:29.443318 9600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:29.443506 9600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:29.450004 9600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35197
I20260619 16:36:29.450016 9666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35197 every 8 connection(s)
I20260619 16:36:29.450868 9600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:29.452251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9600
I20260619 16:36:29.452370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:36:29.452663 392 external_mini_cluster.cc:1428] Setting key 1636c8d9802569a1ccc2975173e91a6b
I20260619 16:36:29.454362 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:29.460719 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429: Bootstrap starting.
I20260619 16:36:29.462726 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:29.454010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.0.98.62:35197)
0619 16:36:29.454628 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:29.454647 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:29.454686 (+ 39us) server_negotiation.cc:407] Connection header received
0619 16:36:29.455498 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:29.455518 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:29.455967 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:29.456306 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:29.456716 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.458167 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.459015 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.459705 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.459875 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.460739 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:29.460746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:29.460762 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:29.461213 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:29.461274 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:29.461299 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:29.461414 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:29.461586 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:36:29.461837 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260619 16:36:29.463303 9667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:29.464244 9667 log.cc:824] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:29.466509 9667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429: No bootstrap required, opened a new log
I20260619 16:36:29.470115 9667 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32a032e8f04be4a78ab542bb62b429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260619 16:36:29.470351 9667 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:29.470420 9667 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc32a032e8f04be4a78ab542bb62b429, State: Initialized, Role: FOLLOWER
I20260619 16:36:29.470875 9667 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32a032e8f04be4a78ab542bb62b429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260619 16:36:29.471000 9667 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:29.471081 9667 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:29.471225 9667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:29.472241 9667 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32a032e8f04be4a78ab542bb62b429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260619 16:36:29.472625 9667 leader_election.cc:302] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc32a032e8f04be4a78ab542bb62b429; no voters:
I20260619 16:36:29.472942 9667 leader_election.cc:288] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:36:29.473104 9672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:29.473373 9672 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [term 1 LEADER]: Becoming Leader. State: Replica: bc32a032e8f04be4a78ab542bb62b429, State: Running, Role: LEADER
I20260619 16:36:29.473697 9672 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32a032e8f04be4a78ab542bb62b429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260619 16:36:29.474201 9667 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:29.475293 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc32a032e8f04be4a78ab542bb62b429. Latest consensus state: current_term: 1 leader_uuid: "bc32a032e8f04be4a78ab542bb62b429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32a032e8f04be4a78ab542bb62b429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } } }
I20260619 16:36:29.475435 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:29.475821 9681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:29.475775 9673 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc32a032e8f04be4a78ab542bb62b429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc32a032e8f04be4a78ab542bb62b429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } } }
I20260619 16:36:29.476428 9673 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:29.478523 9681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:29.483979 9681 catalog_manager.cc:1347] Generated new cluster ID: 9040c770e86d4cb7b28a0f6155a83272
I20260619 16:36:29.484052 9681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:29.496276 9681 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:36:29.497485 9681 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:29.505254 9681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc32a032e8f04be4a78ab542bb62b429: Generated new TSK 0
I20260619 16:36:29.506052 9681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:36:29.516219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 16:36:29.624502 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:29.624782 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:29.624859 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:29.628882 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:29.628960 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:29.629046 9691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:29.633817 9691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:29.634822 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:29.636211 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:36:29.645517 9691 server_base.cc:1034] running on GCE node
W20260619 16:36:29.645485 9699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:29.645503 9696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:29.645718 9697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:29.646157 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:29.646823 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:29.648007 9691 hybrid_clock.cc:648] HybridClock initialized: now 1781886989647979 us; error 40 us; skew 500 ppm
I20260619 16:36:29.649869 9691 webserver.cc:460] Webserver started at http://127.0.98.1:38153/ using document root <none> and password file <none>
I20260619 16:36:29.650381 9691 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:29.650437 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:29.650604 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:29.652483 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "8fa7fa6e25ee42b2906126b16d391177"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "73740216b287c8b0e46e0e800c5afa91"
server_key_iv: "806c53b62ded33c82b3b26f62b0e25d6"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.652978 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8fa7fa6e25ee42b2906126b16d391177"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "73740216b287c8b0e46e0e800c5afa91"
server_key_iv: "806c53b62ded33c82b3b26f62b0e25d6"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.656430 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:36:29.658799 9705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.659961 9691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:36:29.660096 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "8fa7fa6e25ee42b2906126b16d391177"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "73740216b287c8b0e46e0e800c5afa91"
server_key_iv: "806c53b62ded33c82b3b26f62b0e25d6"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.660208 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:29.689920 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:29.690606 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:29.690800 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:29.691371 9691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:29.692288 9691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:29.692359 9691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.692441 9691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:29.692477 9691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.700984 9691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38719
I20260619 16:36:29.701025 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38719 every 8 connection(s)
I20260619 16:36:29.701879 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:29.702395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9691
I20260619 16:36:29.702497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:36:29.702807 392 external_mini_cluster.cc:1428] Setting key 595e283c98ade29ace4424aa2670d0bb
I20260619 16:36:29.704784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:36:29.712589 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:29.703715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39741 (local address 127.0.98.62:35197)
0619 16:36:29.703844 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:29.703848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:29.705041 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 16:36:29.706771 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:29.706774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:29.706820 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:29.706892 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:29.708046 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.708625 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.710523 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.710728 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.710803 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.712082 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:29.712085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:29.712094 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:29.712106 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:29.712124 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:29.712132 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:29.712191 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:29.712386 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:36:29.712482 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:36:29.713874 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:29.704191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.1:39741)
0619 16:36:29.704898 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:29.704948 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:29.706538 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:29.707106 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:29.707117 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:29.707227 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:29.707905 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:29.707926 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.708758 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.708763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:29.710382 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:29.710396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.710884 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.710892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:29.711132 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.711769 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:36:29.711773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:29.711833 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:36:29.711921 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:29.712222 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:29.712234 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:29.712245 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 16:36:29.712388 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 16:36:29.712716 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":132,"threads_started":1}
I20260619 16:36:29.715243 9819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260619 16:36:29.715504 9819 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:29.716022 9819 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260619 16:36:29.717514 9625 ts_manager.cc:194] Registered new tserver with Master: 8fa7fa6e25ee42b2906126b16d391177 (127.0.98.1:38719)
I20260619 16:36:29.718626 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39741
W20260619 16:36:29.814859 9822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:29.815147 9822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:29.815230 9822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:29.819653 9822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:29.819736 9822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:29.819816 9822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:29.824658 9822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:29.825695 9822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:29.826968 9822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:29.836431 9830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:29.836441 9828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:29.836512 9827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:29.836651 9822 server_base.cc:1034] running on GCE node
I20260619 16:36:29.837154 9822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:29.837827 9822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:29.839056 9822 hybrid_clock.cc:648] HybridClock initialized: now 1781886989839042 us; error 35 us; skew 500 ppm
I20260619 16:36:29.840855 9822 webserver.cc:460] Webserver started at http://127.0.98.2:36163/ using document root <none> and password file <none>
I20260619 16:36:29.841475 9822 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:29.841563 9822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:29.841759 9822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:29.843617 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "b32d57869d7b48eaaf095efc7ad8cd6a"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "0fe3f2028e3f6f3e1afe16febd915e1f"
server_key_iv: "7f6590aa77df6104f896fef2b234be2f"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.844151 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b32d57869d7b48eaaf095efc7ad8cd6a"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "0fe3f2028e3f6f3e1afe16febd915e1f"
server_key_iv: "7f6590aa77df6104f896fef2b234be2f"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.847616 9822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:36:29.849936 9836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.851143 9822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:29.851269 9822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "b32d57869d7b48eaaf095efc7ad8cd6a"
format_stamp: "Formatted at 2026-06-19 16:36:29 on dist-test-slave-0f28"
server_key: "0fe3f2028e3f6f3e1afe16febd915e1f"
server_key_iv: "7f6590aa77df6104f896fef2b234be2f"
server_key_version: "encryptionkey@0"
I20260619 16:36:29.851374 9822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:29.864620 9822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:29.865200 9822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:29.865427 9822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:29.865995 9822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:29.866834 9822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:29.866905 9822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.866984 9822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:29.867020 9822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:29.875962 9822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42585
I20260619 16:36:29.876029 9949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42585 every 8 connection(s)
I20260619 16:36:29.876837 9822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:29.881069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9822
I20260619 16:36:29.881186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:36:29.881493 392 external_mini_cluster.cc:1428] Setting key 25c9d828a415451430d43cd497bb7435
I20260619 16:36:29.883281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--builtin_ntp_servers=127.0.98.20:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 16:36:29.886286 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:29.878525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36967 (local address 127.0.98.62:35197)
0619 16:36:29.878664 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:29.878668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:29.879710 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 16:36:29.881022 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:29.881026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:29.881089 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:29.881190 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:29.882144 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.882732 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.883983 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.884381 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.884507 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.885839 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:29.885842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:29.885844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:29.885855 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:29.885873 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:29.885881 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:29.885945 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:29.886116 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 16:36:29.886182 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:36:29.887466 9952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:29.878957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.2:36967)
0619 16:36:29.879571 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:29.879610 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:29.880805 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:29.881421 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:29.881428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:29.881506 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:29.882023 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:29.882035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.882855 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.882859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:29.883866 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:29.883875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:29.884583 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:29.884588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:29.884833 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:29.885540 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:36:29.885544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:29.885605 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:36:29.885687 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:29.885967 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:29.885978 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:29.885987 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:36:29.886112 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 16:36:29.886401 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":106,"threads_started":1}
I20260619 16:36:29.888571 9950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260619 16:36:29.888849 9950 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:29.889529 9950 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260619 16:36:29.890625 9625 ts_manager.cc:194] Registered new tserver with Master: b32d57869d7b48eaaf095efc7ad8cd6a (127.0.98.2:42585)
I20260619 16:36:29.891469 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36967
W20260619 16:36:29.991328 9953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:29.991565 9953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:29.991600 9953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:29.995442 9953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:29.995486 9953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:29.995534 9953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:29.999935 9953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:30.000794 9953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:30.002071 9953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:30.011456 9961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.011648 9953 server_base.cc:1034] running on GCE node
W20260619 16:36:30.011471 9958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:30.011466 9959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.012153 9953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:30.012745 9953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:30.013926 9953 hybrid_clock.cc:648] HybridClock initialized: now 1781886990013912 us; error 24 us; skew 500 ppm
I20260619 16:36:30.015795 9953 webserver.cc:460] Webserver started at http://127.0.98.3:36809/ using document root <none> and password file <none>
I20260619 16:36:30.016299 9953 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:30.016355 9953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:30.016516 9953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:30.018443 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "30b9bc60c3c249acb2c36280fe6e5e55"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "05948406e3459f419a967afc76a78941"
server_key_iv: "b5663e080a49375cc56afd1caca5510f"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.018959 9953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "30b9bc60c3c249acb2c36280fe6e5e55"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "05948406e3459f419a967afc76a78941"
server_key_iv: "b5663e080a49375cc56afd1caca5510f"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.022929 9953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:36:30.025369 9967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.026572 9953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:30.026669 9953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "30b9bc60c3c249acb2c36280fe6e5e55"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "05948406e3459f419a967afc76a78941"
server_key_iv: "b5663e080a49375cc56afd1caca5510f"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.026750 9953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:30.074609 9953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:30.075587 9953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:30.075861 9953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:30.076663 9953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:30.078074 9953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:30.078147 9953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.078212 9953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:30.078250 9953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.090965 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42601 every 8 connection(s)
I20260619 16:36:30.090963 9953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42601
I20260619 16:36:30.092303 9953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:30.100445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 9953
I20260619 16:36:30.100572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:36:30.100812 392 external_mini_cluster.cc:1428] Setting key 2fbeae2cc96fb56bb0bc50d65c8da36b
I20260619 16:36:30.104058 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.094314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33941 (local address 127.0.98.62:35197)
0619 16:36:30.094483 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.094490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.095717 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 16:36:30.097895 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.097898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.097946 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.098024 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.099342 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:30.099890 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:30.101875 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:30.102105 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:30.102187 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:30.103612 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:30.103616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:30.103617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:30.103628 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:30.103646 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:30.103653 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:30.103720 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:30.103905 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 16:36:30.103954 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:36:30.105372 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.094705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.3:33941)
0619 16:36:30.095531 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:30.095590 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:30.097639 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:30.098228 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:30.098239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:30.098382 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:30.099166 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:30.099190 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:30.100024 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:30.100030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:30.101733 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:30.101749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:30.102256 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:30.102262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:30.102510 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:30.103273 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:36:30.103277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:30.103347 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:36:30.103436 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:30.103744 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:30.103754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:30.103762 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:36:30.103898 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 16:36:30.104219 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260619 16:36:30.106544 10081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260619 16:36:30.106775 10081 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:30.107254 10081 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260619 16:36:30.108318 9625 ts_manager.cc:194] Registered new tserver with Master: 30b9bc60c3c249acb2c36280fe6e5e55 (127.0.98.3:42601)
I20260619 16:36:30.108871 9625 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33941
I20260619 16:36:30.114895 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:36:30.117826 9670 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:30.116830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.0.98.62:35197)
0619 16:36:30.116975 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.116982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.117120 (+ 138us) server_negotiation.cc:407] Connection header received
0619 16:36:30.117203 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.117208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.117266 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.117426 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.117685 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44216: BlockingRecv error: recv got EOF from 127.0.0.1:44216 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:36:30.117844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9691
I20260619 16:36:30.124591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9822
I20260619 16:36:30.129804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9953
I20260619 16:36:30.135291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 9600
2026-06-19T16:36:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (875 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T16:36:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:30Z Disabled control of system clock
I20260619 16:36:30.163807 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39913
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:36:30.269160 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:30.269496 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:30.269539 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:30.273461 10092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:30.273509 10092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:30.273524 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:30.273541 10092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:30.273556 10092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:30.278187 10092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:30.279198 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:30.280478 10092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:30.288700 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.288769 10092 server_base.cc:1034] running on GCE node
W20260619 16:36:30.288779 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:30.288730 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.289664 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:30.290719 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:30.291908 10092 hybrid_clock.cc:648] HybridClock initialized: now 1781886990291883 us; error 47 us; skew 500 ppm
I20260619 16:36:30.293854 10092 webserver.cc:460] Webserver started at http://127.0.98.62:33081/ using document root <none> and password file <none>
I20260619 16:36:30.294426 10092 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:30.294519 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:30.294742 10092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:30.296722 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "8e1c051154524c1fbc0951a2c4a6184f"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "19d37d6edb788e142bd2d09699130d39"
server_key_iv: "f5989cac71233f630dd891fdc1706b30"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.297274 10092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e1c051154524c1fbc0951a2c4a6184f"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "19d37d6edb788e142bd2d09699130d39"
server_key_iv: "f5989cac71233f630dd891fdc1706b30"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.300740 10092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:36:30.303156 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.304323 10092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:30.304478 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "8e1c051154524c1fbc0951a2c4a6184f"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "19d37d6edb788e142bd2d09699130d39"
server_key_iv: "f5989cac71233f630dd891fdc1706b30"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.304592 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:30.309158 10092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:30.309823 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:30.310009 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:30.316787 10092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39913
I20260619 16:36:30.316813 10158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39913 every 8 connection(s)
I20260619 16:36:30.317754 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:30.318729 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10092
I20260619 16:36:30.318879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:36:30.319182 392 external_mini_cluster.cc:1428] Setting key 33f95744f152a43e01f8fabcb3392713
I20260619 16:36:30.320851 10159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:30.324918 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.320496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.0.98.62:39913)
0619 16:36:30.321584 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.321600 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.321635 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:36:30.322249 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.322263 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.322607 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.322878 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.323681 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:30.323685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:30.323695 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:30.323704 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:30.323754 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:30.323770 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:30.323876 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:30.323978 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 16:36:30.324235 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":599,"threads_started":1}
I20260619 16:36:30.327127 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f: Bootstrap starting.
I20260619 16:36:30.329124 10159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:30.329861 10159 log.cc:824] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:30.331761 10159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f: No bootstrap required, opened a new log
I20260619 16:36:30.335487 10159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1c051154524c1fbc0951a2c4a6184f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260619 16:36:30.335722 10159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:30.335812 10159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1c051154524c1fbc0951a2c4a6184f, State: Initialized, Role: FOLLOWER
I20260619 16:36:30.336215 10159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1c051154524c1fbc0951a2c4a6184f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260619 16:36:30.336357 10159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:30.336442 10159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:30.336556 10159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:30.337435 10159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1c051154524c1fbc0951a2c4a6184f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260619 16:36:30.337755 10159 leader_election.cc:302] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e1c051154524c1fbc0951a2c4a6184f; no voters:
I20260619 16:36:30.338092 10159 leader_election.cc:288] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:36:30.338209 10164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:30.338442 10164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [term 1 LEADER]: Becoming Leader. State: Replica: 8e1c051154524c1fbc0951a2c4a6184f, State: Running, Role: LEADER
I20260619 16:36:30.338765 10164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1c051154524c1fbc0951a2c4a6184f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } }
I20260619 16:36:30.339298 10159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:30.340406 10166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e1c051154524c1fbc0951a2c4a6184f. Latest consensus state: current_term: 1 leader_uuid: "8e1c051154524c1fbc0951a2c4a6184f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1c051154524c1fbc0951a2c4a6184f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } } }
I20260619 16:36:30.340531 10166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:30.340857 10165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e1c051154524c1fbc0951a2c4a6184f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1c051154524c1fbc0951a2c4a6184f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39913 } } }
I20260619 16:36:30.340975 10173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:30.341006 10165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:30.344054 10173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:30.349372 10173 catalog_manager.cc:1347] Generated new cluster ID: b758ddb252cd4f019130c0527b842e1c
I20260619 16:36:30.349460 10173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:30.362488 10173 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:36:30.362838 10173 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:30.376400 10173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e1c051154524c1fbc0951a2c4a6184f: Generated new TSK 0
I20260619 16:36:30.377164 10173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:36:30.390605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39913
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 16:36:30.500830 10183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:30.501091 10183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:30.501132 10183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:30.505053 10183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:30.505100 10183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:30.505157 10183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:30.510032 10183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:30.510977 10183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:30.512244 10183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:30.521518 10188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:30.521637 10189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.521773 10183 server_base.cc:1034] running on GCE node
W20260619 16:36:30.521608 10191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.522210 10183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:30.522795 10183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:30.523985 10183 hybrid_clock.cc:648] HybridClock initialized: now 1781886990523972 us; error 24 us; skew 500 ppm
I20260619 16:36:30.525908 10183 webserver.cc:460] Webserver started at http://127.0.98.1:35661/ using document root <none> and password file <none>
I20260619 16:36:30.526468 10183 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:30.526556 10183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:30.526783 10183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:30.528716 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "7bf7eca5ef354a87bec11560377decd5"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "5336676c9e54f2266dfd4a087eeff74d"
server_key_iv: "52589838a31735a3863af06ce861ea91"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.529260 10183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7bf7eca5ef354a87bec11560377decd5"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "5336676c9e54f2266dfd4a087eeff74d"
server_key_iv: "52589838a31735a3863af06ce861ea91"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.532718 10183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:36:30.535153 10197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.536386 10183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:36:30.536523 10183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "7bf7eca5ef354a87bec11560377decd5"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "5336676c9e54f2266dfd4a087eeff74d"
server_key_iv: "52589838a31735a3863af06ce861ea91"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.536629 10183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:30.541090 10183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:30.541692 10183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:30.541865 10183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:30.542363 10183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:30.543186 10183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:30.543254 10183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.543314 10183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:30.543365 10183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.552119 10183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33905
I20260619 16:36:30.552143 10310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33905 every 8 connection(s)
I20260619 16:36:30.552908 10183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:36:30.556414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10183
I20260619 16:36:30.556504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:36:30.556762 392 external_mini_cluster.cc:1428] Setting key 791c4d46b47ed80c47d7602254c5dd67
I20260619 16:36:30.558781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39913
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:36:30.558873 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.554539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41615 (local address 127.0.98.62:39913)
0619 16:36:30.554681 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.554685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.555718 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 16:36:30.557079 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.557082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.557129 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.557212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.558429 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:30.558432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:30.558434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:30.558444 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:30.558461 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:30.558468 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:30.558540 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:30.558717 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:36:30.558751 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:36:30.559896 10313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.554972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39913 (local address 127.0.98.1:41615)
0619 16:36:30.555577 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:30.555615 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:30.556883 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:30.557430 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:30.557437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:30.557519 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:30.558143 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:36:30.558145 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:30.558196 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:36:30.558274 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:30.558557 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:30.558569 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:30.558578 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 16:36:30.558747 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 16:36:30.558979 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260619 16:36:30.561064 10311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39913
I20260619 16:36:30.561362 10311 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:30.561746 10311 heartbeater.cc:511] Master 127.0.98.62:39913 requested a full tablet report, sending...
I20260619 16:36:30.563086 10123 ts_manager.cc:194] Registered new tserver with Master: 7bf7eca5ef354a87bec11560377decd5 (127.0.98.1:33905)
W20260619 16:36:30.666574 10314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:30.666841 10314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:30.666913 10314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:30.670827 10314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:30.670905 10314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:30.670991 10314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:30.675747 10314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:30.676746 10314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:30.678099 10314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:30.687439 10322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:30.687449 10320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.687471 10314 server_base.cc:1034] running on GCE node
W20260619 16:36:30.687440 10319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.688037 10314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:30.688618 10314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:30.689851 10314 hybrid_clock.cc:648] HybridClock initialized: now 1781886990689817 us; error 45 us; skew 500 ppm
I20260619 16:36:30.691679 10314 webserver.cc:460] Webserver started at http://127.0.98.2:45767/ using document root <none> and password file <none>
I20260619 16:36:30.692230 10314 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:30.692327 10314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:30.692552 10314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:30.694705 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "39a87dbc344c43a686344d238d53cbbd"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "ab8b905796575f26b56b53afc68ad150"
server_key_iv: "505562e037f1926d06ffe78f8fb44996"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.695258 10314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "39a87dbc344c43a686344d238d53cbbd"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "ab8b905796575f26b56b53afc68ad150"
server_key_iv: "505562e037f1926d06ffe78f8fb44996"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.698745 10314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:36:30.701073 10328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.702240 10314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:36:30.702431 10314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "39a87dbc344c43a686344d238d53cbbd"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "ab8b905796575f26b56b53afc68ad150"
server_key_iv: "505562e037f1926d06ffe78f8fb44996"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.702550 10314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:30.706993 10314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:30.707544 10314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:30.707717 10314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:30.708247 10314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:30.709123 10314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:30.709189 10314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.709249 10314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:30.709298 10314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.718644 10314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43267
I20260619 16:36:30.718662 10441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43267 every 8 connection(s)
I20260619 16:36:30.719544 10314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:30.725225 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.721365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45175 (local address 127.0.98.62:39913)
0619 16:36:30.721523 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.721528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.722525 (+ 997us) server_negotiation.cc:407] Connection header received
0619 16:36:30.723700 (+ 1175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.723703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.723739 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.723813 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.724785 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:30.724788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:30.724790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:30.724801 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:30.724820 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:30.724836 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:30.724912 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:30.725083 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 16:36:30.725118 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:36:30.725865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10314
I20260619 16:36:30.725996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:36:30.726166 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.721756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39913 (local address 127.0.98.2:45175)
0619 16:36:30.722375 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:30.722415 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:30.723522 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:30.723961 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:30.723967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:30.724048 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:30.724550 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:36:30.724552 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:30.724589 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:36:30.724651 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:30.724941 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:30.724948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:30.724953 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:36:30.725097 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 16:36:30.725270 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260619 16:36:30.726351 392 external_mini_cluster.cc:1428] Setting key 81a1ba7dbc7d750c9f417985eca0fb7a
I20260619 16:36:30.727291 10442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39913
I20260619 16:36:30.727547 10442 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:30.727914 10442 heartbeater.cc:511] Master 127.0.98.62:39913 requested a full tablet report, sending...
I20260619 16:36:30.728230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39913
--builtin_ntp_servers=127.0.98.20:35773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 16:36:30.728888 10123 ts_manager.cc:194] Registered new tserver with Master: 39a87dbc344c43a686344d238d53cbbd (127.0.98.2:43267)
W20260619 16:36:30.835279 10445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:30.835515 10445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:30.835553 10445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:30.839601 10445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:30.839668 10445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:30.839759 10445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:30.844836 10445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:30.845830 10445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:30.847054 10445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:30.856323 10453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:30.856328 10451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:30.856328 10450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:30.857196 10445 server_base.cc:1034] running on GCE node
I20260619 16:36:30.857600 10445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:30.858186 10445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:30.859390 10445 hybrid_clock.cc:648] HybridClock initialized: now 1781886990859364 us; error 30 us; skew 500 ppm
I20260619 16:36:30.861191 10445 webserver.cc:460] Webserver started at http://127.0.98.3:37281/ using document root <none> and password file <none>
I20260619 16:36:30.862015 10445 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:30.862118 10445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:30.862358 10445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:30.864418 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "f066ceed70ca4d509ed7f449dbd60f55"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "e2cd212ec5940b31dadc0b74a8a6d192"
server_key_iv: "168c7dab81cfbc290a9732ff2eb422ed"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.864950 10445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f066ceed70ca4d509ed7f449dbd60f55"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "e2cd212ec5940b31dadc0b74a8a6d192"
server_key_iv: "168c7dab81cfbc290a9732ff2eb422ed"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.870088 10445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 16:36:30.872409 10459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.873770 10445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:30.873905 10445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f066ceed70ca4d509ed7f449dbd60f55"
format_stamp: "Formatted at 2026-06-19 16:36:30 on dist-test-slave-0f28"
server_key: "e2cd212ec5940b31dadc0b74a8a6d192"
server_key_iv: "168c7dab81cfbc290a9732ff2eb422ed"
server_key_version: "encryptionkey@0"
I20260619 16:36:30.874012 10445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:30.878703 10445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:30.879266 10445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:30.879443 10445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:30.879954 10445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:30.880810 10445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:30.880878 10445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.880939 10445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:30.880988 10445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:30.890048 10445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38267
I20260619 16:36:30.890077 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38267 every 8 connection(s)
I20260619 16:36:30.890887 10445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:30.894068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10445
I20260619 16:36:30.894171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:36:30.894454 392 external_mini_cluster.cc:1428] Setting key c8e70b04efbe211bf0f6215e828cfbb8
I20260619 16:36:30.898007 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.893246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57637 (local address 127.0.98.62:39913)
0619 16:36:30.893414 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.893418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.894646 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 16:36:30.896386 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.896392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.896476 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.896559 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.897643 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:30.897646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:30.897648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:36:30.897658 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:30.897677 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:36:30.897685 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:30.897757 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:30.897875 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 16:36:30.897909 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:36:30.898859 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:30.893707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39913 (local address 127.0.98.3:57637)
0619 16:36:30.894470 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:30.894534 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:30.896122 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:30.896746 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:30.896753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:30.896840 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:30.897392 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:36:30.897394 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:30.897444 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:36:30.897510 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:30.897760 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:30.897766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:30.897770 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:36:30.897897 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:36:30.898129 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":137,"threads_started":1}
I20260619 16:36:30.899941 10573 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39913
I20260619 16:36:30.900138 10573 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:30.900477 10573 heartbeater.cc:511] Master 127.0.98.62:39913 requested a full tablet report, sending...
I20260619 16:36:30.901415 10123 ts_manager.cc:194] Registered new tserver with Master: f066ceed70ca4d509ed7f449dbd60f55 (127.0.98.3:38267)
I20260619 16:36:30.908209 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 16:36:30.910894 10162 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:36:30.910190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.0.98.62:39913)
0619 16:36:30.910316 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:30.910319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:30.910448 (+ 129us) server_negotiation.cc:407] Connection header received
0619 16:36:30.910542 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:30.910544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:30.910569 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:30.910633 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:30.910808 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52998: BlockingRecv error: recv got EOF from 127.0.0.1:52998 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:36:30.911195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10183
I20260619 16:36:30.916522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10314
I20260619 16:36:30.921453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10445
I20260619 16:36:30.926661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10092
2026-06-19T16:36:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (791 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:36:30 dist-test-slave-0f28 krb5kdc[10583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:36:30 dist-test-slave-0f28 krb5kdc[10583](info): set up 2 sockets
Jun 19 16:36:30 dist-test-slave-0f28 krb5kdc[10583](info): commencing operation
krb5kdc: starting...
W20260619 16:36:31.962344 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:36:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:36:32.082397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38231
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:32.190224 10596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:32.190507 10596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:32.190586 10596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:32.194485 10596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:36:32.194563 10596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:32.194608 10596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:32.194665 10596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:36:32.194703 10596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:36:32.199882 10596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:32.200832 10596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:32.201812 10596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:32.209918 10604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:32.209981 10596 server_base.cc:1034] running on GCE node
W20260619 16:36:32.210106 10601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.209993 10602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:32.210619 10596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:32.211642 10596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:32.212796 10596 hybrid_clock.cc:648] HybridClock initialized: now 1781886992212785 us; error 27 us; skew 500 ppm
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:32.215430 10596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:36:32.216240 10596 webserver.cc:460] Webserver started at http://127.0.98.62:39155/ using document root <none> and password file <none>
I20260619 16:36:32.216774 10596 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:32.216874 10596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:32.217094 10596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:32.219062 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "074a68eac28f449d9a901e5c6fbcdaa3"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "9eb33eda05143a018d53ecf4e03e630f"
server_key_iv: "f80fae36d77eacd4382284e82118b891"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.219596 10596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "074a68eac28f449d9a901e5c6fbcdaa3"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "9eb33eda05143a018d53ecf4e03e630f"
server_key_iv: "f80fae36d77eacd4382284e82118b891"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.223095 10596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:36:32.225306 10611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.226521 10596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:36:32.226665 10596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "074a68eac28f449d9a901e5c6fbcdaa3"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "9eb33eda05143a018d53ecf4e03e630f"
server_key_iv: "f80fae36d77eacd4382284e82118b891"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.226771 10596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:32.247809 10596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:32.250540 10596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:32.250741 10596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:32.257232 10596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38231
I20260619 16:36:32.257350 10663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38231 every 8 connection(s)
I20260619 16:36:32.258239 10596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:36:32.260828 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:32.266033 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3: Bootstrap starting.
I20260619 16:36:32.268075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10596
I20260619 16:36:32.268173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:36:32.268462 392 external_mini_cluster.cc:1428] Setting key b49914f02f3e102ba779c6deca144925
I20260619 16:36:32.268460 10664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:32.269433 10664 log.cc:824] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:32.271353 10664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3: No bootstrap required, opened a new log
I20260619 16:36:32.275228 10664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38231 } }
I20260619 16:36:32.275503 10664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:32.275579 10664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074a68eac28f449d9a901e5c6fbcdaa3, State: Initialized, Role: FOLLOWER
I20260619 16:36:32.276077 10664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38231 } }
I20260619 16:36:32.276257 10664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:36:32.276320 10664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:36:32.276414 10664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:36:32.277307 10664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38231 } }
I20260619 16:36:32.277670 10664 leader_election.cc:302] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 074a68eac28f449d9a901e5c6fbcdaa3; no voters:
I20260619 16:36:32.277961 10664 leader_election.cc:288] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:36:32.278069 10669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:32.278355 10669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [term 1 LEADER]: Becoming Leader. State: Replica: 074a68eac28f449d9a901e5c6fbcdaa3, State: Running, Role: LEADER
I20260619 16:36:32.278730 10669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38231 } }
I20260619 16:36:32.279292 10664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:36:32.280469 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 074a68eac28f449d9a901e5c6fbcdaa3. Latest consensus state: current_term: 1 leader_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38231 } } }
I20260619 16:36:32.280591 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:32.280989 10679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:36:32.281646 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "074a68eac28f449d9a901e5c6fbcdaa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38231 } } }
I20260619 16:36:32.281762 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3 [sys.catalog]: This master's current role is: LEADER
I20260619 16:36:32.283910 10679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:36:32.284108 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.269551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.98.62:38231)
0619 16:36:32.269990 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:32.270005 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:32.270038 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:36:32.270662 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:32.270678 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:32.270980 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:32.271240 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:32.272095 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.273059 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.274345 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.274795 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.274906 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.277693 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:32.277698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:32.277712 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:32.277738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:32.279858 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.280901 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.280908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.280913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.281052 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.281939 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.281942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.281945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.282433 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:32.282624 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:32.282937 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 16:36:32.283227 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":156,"threads_started":1}
I20260619 16:36:32.289119 10679 catalog_manager.cc:1347] Generated new cluster ID: dc669f434d134cdab0cd7f42a2ef775f
I20260619 16:36:32.289179 10679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:36:32.304407 10679 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:36:32.305630 10679 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:36:32.315953 10679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3: Generated new TSK 0
I20260619 16:36:32.316649 10679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:36:32.348340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38231
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:32.458381 10690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:32.458602 10690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:32.458642 10690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:32.462572 10690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:32.462625 10690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:32.462718 10690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:36:32.467695 10690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:32.468619 10690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:32.469630 10690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:32.478731 10698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.478765 10696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.478725 10695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:32.479080 10690 server_base.cc:1034] running on GCE node
I20260619 16:36:32.479529 10690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:32.480305 10690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:32.481518 10690 hybrid_clock.cc:648] HybridClock initialized: now 1781886992481499 us; error 30 us; skew 500 ppm
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:32.484308 10690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:36:32.485131 10690 webserver.cc:460] Webserver started at http://127.0.98.1:42161/ using document root <none> and password file <none>
I20260619 16:36:32.485718 10690 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:32.485805 10690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:32.486023 10690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:32.487926 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "7e978e744bf54de6b8d63a5ca82c0afe"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "604e279d595d330db5ef72158d130b59"
server_key_iv: "1ed2fc88724e330ad72c5118317be559"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.488471 10690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7e978e744bf54de6b8d63a5ca82c0afe"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "604e279d595d330db5ef72158d130b59"
server_key_iv: "1ed2fc88724e330ad72c5118317be559"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.491953 10690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:36:32.494323 10705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.495455 10690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:32.495579 10690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "7e978e744bf54de6b8d63a5ca82c0afe"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "604e279d595d330db5ef72158d130b59"
server_key_iv: "1ed2fc88724e330ad72c5118317be559"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.495679 10690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:32.510185 10690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:32.513141 10690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:32.513417 10690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:32.513937 10690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:32.514667 10690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:32.514734 10690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.514798 10690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:32.514846 10690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.523289 10690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40401
I20260619 16:36:32.523311 10818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40401 every 8 connection(s)
I20260619 16:36:32.524122 10690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:36:32.533849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10690
I20260619 16:36:32.533974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:36:32.534250 392 external_mini_cluster.cc:1428] Setting key 4a640db7737719279fc5583fa7392173
I20260619 16:36:32.537420 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.525846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46455 (local address 127.0.98.62:38231)
0619 16:36:32.525971 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:32.525974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:32.526830 (+ 856us) server_negotiation.cc:407] Connection header received
0619 16:36:32.528035 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:32.528038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:32.528094 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:32.528196 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:32.529274 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.529931 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.531132 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.531385 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.531498 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.533790 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:32.533793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:32.533795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:32.533817 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:32.535451 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.535949 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.535953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.535954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.536039 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.536372 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.536375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.536377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.536553 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:32.536656 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:32.537109 (+ 453us) server_negotiation.cc:299] Negotiation successful
0619 16:36:32.537223 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:36:32.538267 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.526251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38231 (local address 127.0.98.1:46455)
0619 16:36:32.526685 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:32.526723 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:32.527831 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:32.528352 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:32.528359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:32.528705 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:32.529128 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:32.529141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.530086 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.530089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:32.531015 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:32.531024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.531529 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.531533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:32.531691 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.532158 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:32.532173 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:32.533627 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:32.535573 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:32.535579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:32.535591 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:32.535827 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:32.536135 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:32.536138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:32.536140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:32.536271 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:32.536655 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:32.536662 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:32.536885 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 16:36:32.537070 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 16:36:32.537380 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":129,"threads_started":1}
I20260619 16:36:32.539590 10819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38231
I20260619 16:36:32.539826 10819 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:32.540266 10819 heartbeater.cc:511] Master 127.0.98.62:38231 requested a full tablet report, sending...
I20260619 16:36:32.541796 10628 ts_manager.cc:194] Registered new tserver with Master: 7e978e744bf54de6b8d63a5ca82c0afe (127.0.98.1:40401)
I20260619 16:36:32.542901 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46455
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:36:32.560976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38231
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:32.669411 10824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:32.669701 10824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:32.669786 10824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:32.673753 10824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:32.673874 10824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:32.673972 10824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:36:32.679105 10824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:32.680106 10824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:32.681131 10824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:32.690629 10829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.690723 10832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.690722 10830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:32.690905 10824 server_base.cc:1034] running on GCE node
I20260619 16:36:32.691481 10824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:32.692128 10824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:32.693356 10824 hybrid_clock.cc:648] HybridClock initialized: now 1781886992693304 us; error 27 us; skew 500 ppm
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:32.696086 10824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:36:32.696935 10824 webserver.cc:460] Webserver started at http://127.0.98.2:36533/ using document root <none> and password file <none>
I20260619 16:36:32.697528 10824 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:32.697614 10824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:32.697829 10824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:32.699900 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "0550f83249d14e25ab0c593e60fabee7"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "1efa114d23d3aa2aac5ee8897e773a07"
server_key_iv: "f4aed21ecd0cbbfb908fcf6ab3602ed8"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.700434 10824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0550f83249d14e25ab0c593e60fabee7"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "1efa114d23d3aa2aac5ee8897e773a07"
server_key_iv: "f4aed21ecd0cbbfb908fcf6ab3602ed8"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.703974 10824 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:36:32.706319 10839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.707571 10824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:32.707667 10824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "0550f83249d14e25ab0c593e60fabee7"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "1efa114d23d3aa2aac5ee8897e773a07"
server_key_iv: "f4aed21ecd0cbbfb908fcf6ab3602ed8"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.707746 10824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:32.728008 10824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:32.730849 10824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:32.731047 10824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:32.731631 10824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:32.732373 10824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:32.732420 10824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.732458 10824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:32.732474 10824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.741484 10824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32855
I20260619 16:36:32.741523 10952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32855 every 8 connection(s)
I20260619 16:36:32.742312 10824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:36:32.746778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10824
I20260619 16:36:32.746898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:36:32.747200 392 external_mini_cluster.cc:1428] Setting key 34d03b6709f980008674c2a3545d102d
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:36:32.756532 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.744070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41463 (local address 127.0.98.62:38231)
0619 16:36:32.744237 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:32.744243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:32.745069 (+ 826us) server_negotiation.cc:407] Connection header received
0619 16:36:32.746320 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:32.746323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:32.746376 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:32.746454 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:32.747841 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.748366 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.749501 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.749721 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.749829 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.752375 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:32.752382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:32.752388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:32.752420 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:32.754329 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.754891 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.754896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.754899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.754997 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.755333 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.755337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.755339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.755583 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:32.755695 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:32.756207 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 16:36:32.756335 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:36:32.757251 10955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.744517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38231 (local address 127.0.98.2:41463)
0619 16:36:32.744929 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:32.744968 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:32.746120 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:32.746611 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:32.746622 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:32.747095 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:32.747704 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:32.747720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.748484 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.748489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:32.749382 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:32.749398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.749860 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.749865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:32.750029 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.750592 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:32.750607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:32.752215 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:32.754471 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:32.754476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:32.754487 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:32.754757 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:32.755097 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:32.755100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:32.755101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:32.755235 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:32.755714 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:32.755720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:32.755944 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 16:36:32.756114 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:36:32.756355 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":107,"threads_started":1}
I20260619 16:36:32.758479 10953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38231
I20260619 16:36:32.758716 10953 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:32.759260 10953 heartbeater.cc:511] Master 127.0.98.62:38231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:36:32.760387 10628 ts_manager.cc:194] Registered new tserver with Master: 0550f83249d14e25ab0c593e60fabee7 (127.0.98.2:32855)
I20260619 16:36:32.761217 10628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41463
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:36:32.776895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38231
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:36:32.890211 10958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:36:32.890467 10958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:36:32.890534 10958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:36:32.894564 10958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:36:32.894639 10958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:36:32.894732 10958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:36:32.899914 10958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:36:32.900861 10958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:36:32.901942 10958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:36:32.911060 10963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.911132 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:36:32.911253 10966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:36:32.911258 10958 server_base.cc:1034] running on GCE node
I20260619 16:36:32.911765 10958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:36:32.912402 10958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:36:32.913576 10958 hybrid_clock.cc:648] HybridClock initialized: now 1781886992913569 us; error 28 us; skew 500 ppm
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:36:32.916111 10958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:36:32.916965 10958 webserver.cc:460] Webserver started at http://127.0.98.3:42401/ using document root <none> and password file <none>
I20260619 16:36:32.917516 10958 fs_manager.cc:359] Metadata directory not provided
I20260619 16:36:32.917577 10958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:36:32.917727 10958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:36:32.919535 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "f371941bb5384f44a66951c50f7c6204"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "09349512990daf1757593fce0699c118"
server_key_iv: "1b160f29e248ae6988364f33ad15ace4"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.919997 10958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f371941bb5384f44a66951c50f7c6204"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "09349512990daf1757593fce0699c118"
server_key_iv: "1b160f29e248ae6988364f33ad15ace4"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.923454 10958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 16:36:32.925704 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.926891 10958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:36:32.927016 10958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f371941bb5384f44a66951c50f7c6204"
format_stamp: "Formatted at 2026-06-19 16:36:32 on dist-test-slave-0f28"
server_key: "09349512990daf1757593fce0699c118"
server_key_iv: "1b160f29e248ae6988364f33ad15ace4"
server_key_version: "encryptionkey@0"
I20260619 16:36:32.927116 10958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:36:32.948585 10958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:36:32.951309 10958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:36:32.951503 10958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:36:32.952028 10958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:36:32.952847 10958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:36:32.952891 10958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.952956 10958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:36:32.952991 10958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:36:32.961558 10958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38801
I20260619 16:36:32.961585 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38801 every 8 connection(s)
I20260619 16:36:32.962360 10958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:36:32.962826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 10958
I20260619 16:36:32.962941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:36:32.963219 392 external_mini_cluster.cc:1428] Setting key 231ebf38b327853d7d7315e42cb3eb32
Jun 19 16:36:32 dist-test-slave-0f28 krb5kdc[10583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:36:32.975297 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.964219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34985 (local address 127.0.98.62:38231)
0619 16:36:32.964326 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:32.964329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:32.965271 (+ 942us) server_negotiation.cc:407] Connection header received
0619 16:36:32.966528 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:32.966531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:32.966575 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:32.966652 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:32.967719 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.968259 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.969420 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.969643 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.969722 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.972034 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:32.972038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:32.972040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:32.972062 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:32.973673 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.974112 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.974114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.974115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.974183 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.974454 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.974456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.974457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.974610 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:32.974681 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:32.975093 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 16:36:32.975186 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 16:36:32.976147 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.964691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38231 (local address 127.0.98.3:34985)
0619 16:36:32.965141 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:32.965193 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:32.966329 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:32.966804 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:32.966809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:32.967134 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:36:32.967588 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:32.967601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.968419 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.968421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:32.969232 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:32.969240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.969784 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.969787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:32.969928 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.970432 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:36:32.970447 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:36:32.971885 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:36:32.973772 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:32.973776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:32.973787 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:32.974027 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:32.974258 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:36:32.974260 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:36:32.974261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:36:32.974378 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:36:32.974691 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:36:32.974695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:36:32.974900 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 16:36:32.975043 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 16:36:32.975255 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260619 16:36:32.977210 11087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38231
I20260619 16:36:32.977485 11087 heartbeater.cc:464] Registering TS with master...
I20260619 16:36:32.977876 11087 heartbeater.cc:511] Master 127.0.98.62:38231 requested a full tablet report, sending...
I20260619 16:36:32.978746 10627 ts_manager.cc:194] Registered new tserver with Master: f371941bb5384f44a66951c50f7c6204 (127.0.98.3:38801)
I20260619 16:36:32.979331 10627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34985
I20260619 16:36:32.988471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:36:33.000664 10667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:32.990250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.98.62:38231)
0619 16:36:32.990359 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:32.990363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:32.990531 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:36:32.990607 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:32.990609 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:32.990646 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:32.990709 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:36:32.991320 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.991822 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.992567 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:32.992763 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:32.992834 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:32.997461 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:36:32.997464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:36:32.997466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:36:32.997486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:36:32.999164 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.999539 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.999542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.999543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:32.999621 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:36:32.999855 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:36:32.999858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:36:32.999859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:36:33.000144 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:36:33.000249 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.000466 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.000555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:36:33.003666 10627 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:36:33.006572 10627 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:36:33.020735 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.014568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40401 (local address 127.0.0.1:59508)
0619 16:36:33.014908 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.014954 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:33.015084 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:33.016464 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:33.016467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:33.016483 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:33.016591 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.016598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.017987 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.017990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.019234 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.019243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.020197 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.020201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.020326 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.020372 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 16:36:33.020472 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:36:33.020580 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":64,"thread_start_us":74,"threads_started":1}
I20260619 16:36:33.020956 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.015603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.0.98.2:32855)
0619 16:36:33.015904 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:33.015913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:33.015931 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:36:33.016023 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:33.016028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:33.016182 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:33.016332 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:33.016606 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.017852 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.019718 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.020337 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.020454 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.020586 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.020698 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.020800 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260619 16:36:33.021126 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.015122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38801 (local address 127.0.0.1:48752)
0619 16:36:33.015349 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.015361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:33.015444 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:33.017101 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:33.017104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:33.017121 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:33.017208 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.017214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.018367 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.018370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.019554 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.019564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.020742 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.020745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.020831 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.020850 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 16:36:33.020917 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 16:36:33.020986 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
I20260619 16:36:33.021543 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.014715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.0.98.1:40401)
0619 16:36:33.016109 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:33.016122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:33.016140 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:36:33.016195 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:33.016198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:33.016338 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:33.016489 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:33.016713 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.017860 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.019352 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.020070 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.020177 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.020315 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.021348 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.021437 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1310,"thread_start_us":80,"threads_started":1}
I20260619 16:36:33.021564 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.015300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.0.98.3:38801)
0619 16:36:33.016775 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:33.016783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:33.016799 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:36:33.016854 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:33.016857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:33.016972 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:33.017111 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:33.017382 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.018209 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.019676 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.020461 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.020559 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.020688 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.021280 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.021408 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1371,"thread_start_us":176,"threads_started":1}
I20260619 16:36:33.020917 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.015238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32855 (local address 127.0.0.1:35074)
0619 16:36:33.015636 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.015649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:33.015757 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:33.016332 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:33.016338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:33.016358 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:33.016483 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.016490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.018504 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.018507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.019586 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.019596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.020443 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.020447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.020565 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.020587 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:36:33.020657 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 16:36:33.020732 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":87,"threads_started":1}
I20260619 16:36:33.023916 10753 tablet_service.cc:1467] Processing CreateTablet for tablet 48a27c9c33374c6092ab8db3342d3223 (DEFAULT_TABLE table=test-table [id=f82890c232f74f4da95420448dc625db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:33.024765 10753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a27c9c33374c6092ab8db3342d3223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:33.027220 10887 tablet_service.cc:1467] Processing CreateTablet for tablet 48a27c9c33374c6092ab8db3342d3223 (DEFAULT_TABLE table=test-table [id=f82890c232f74f4da95420448dc625db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:33.027559 11021 tablet_service.cc:1467] Processing CreateTablet for tablet 48a27c9c33374c6092ab8db3342d3223 (DEFAULT_TABLE table=test-table [id=f82890c232f74f4da95420448dc625db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:36:33.028165 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a27c9c33374c6092ab8db3342d3223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:33.028473 11021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a27c9c33374c6092ab8db3342d3223. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:36:33.031944 11104 tablet_bootstrap.cc:492] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe: Bootstrap starting.
I20260619 16:36:33.034600 11105 tablet_bootstrap.cc:492] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Bootstrap starting.
I20260619 16:36:33.035082 11104 tablet_bootstrap.cc:654] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:33.035343 11106 tablet_bootstrap.cc:492] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: Bootstrap starting.
I20260619 16:36:33.036455 11104 log.cc:824] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:33.036767 11105 tablet_bootstrap.cc:654] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:33.037598 11105 log.cc:824] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:33.037806 11106 tablet_bootstrap.cc:654] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: Neither blocks nor log segments found. Creating new log.
I20260619 16:36:33.038621 11104 tablet_bootstrap.cc:492] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe: No bootstrap required, opened a new log
I20260619 16:36:33.038568 11106 log.cc:824] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: Log is configured to *not* fsync() on all Append() calls
I20260619 16:36:33.038897 11104 ts_tablet_manager.cc:1397] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:36:33.039539 11105 tablet_bootstrap.cc:492] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: No bootstrap required, opened a new log
I20260619 16:36:33.039731 11105 ts_tablet_manager.cc:1397] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:36:33.041225 11106 tablet_bootstrap.cc:492] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: No bootstrap required, opened a new log
I20260619 16:36:33.041471 11106 ts_tablet_manager.cc:1397] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 16:36:33.042523 11105 raft_consensus.cc:348] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.042574 11104 raft_consensus.cc:348] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.042760 11105 raft_consensus.cc:374] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:33.042809 11105 raft_consensus.cc:729] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f371941bb5384f44a66951c50f7c6204, State: Initialized, Role: FOLLOWER
I20260619 16:36:33.042835 11104 raft_consensus.cc:374] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:33.042881 11104 raft_consensus.cc:729] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e978e744bf54de6b8d63a5ca82c0afe, State: Initialized, Role: FOLLOWER
I20260619 16:36:33.043262 11105 consensus_queue.cc:260] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.043316 11104 consensus_queue.cc:260] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.043852 11087 heartbeater.cc:503] Master 127.0.98.62:38231 was elected leader, sending a full tablet report...
I20260619 16:36:33.044176 11105 ts_tablet_manager.cc:1428] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:36:33.044190 11104 ts_tablet_manager.cc:1428] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:36:33.044482 10819 heartbeater.cc:503] Master 127.0.98.62:38231 was elected leader, sending a full tablet report...
I20260619 16:36:33.045078 11106 raft_consensus.cc:348] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.045399 11106 raft_consensus.cc:374] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:36:33.045512 11106 raft_consensus.cc:729] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0550f83249d14e25ab0c593e60fabee7, State: Initialized, Role: FOLLOWER
I20260619 16:36:33.045935 11106 consensus_queue.cc:260] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.046517 10953 heartbeater.cc:503] Master 127.0.98.62:38231 was elected leader, sending a full tablet report...
I20260619 16:36:33.046859 11106 ts_tablet_manager.cc:1428] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260619 16:36:33.064514 11110 raft_consensus.cc:482] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:36:33.064675 11110 raft_consensus.cc:504] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.065778 11110 leader_election.cc:288] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e978e744bf54de6b8d63a5ca82c0afe (127.0.98.1:40401), 0550f83249d14e25ab0c593e60fabee7 (127.0.98.2:32855)
I20260619 16:36:33.070564 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.065945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40401 (local address 127.0.98.3:44373)
0619 16:36:33.066064 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.066079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:33.066161 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:33.066561 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:33.066564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:33.066582 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:33.066708 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.066715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.067774 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.067776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.069041 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.069052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.070009 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.070013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.070184 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.070252 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 16:36:33.070349 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:36:33.070425 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":45}
I20260619 16:36:33.070583 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.066183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44373 (local address 127.0.98.1:40401)
0619 16:36:33.066304 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:33.066308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:33.066322 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:36:33.066435 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:33.066437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:33.066476 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:33.066542 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:33.066808 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.067613 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.069168 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.069852 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.069970 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.070035 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.070410 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.070469 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:36:33.070583 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.066320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50599 (local address 127.0.98.2:32855)
0619 16:36:33.066459 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:33.066465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:33.066701 (+ 236us) server_negotiation.cc:407] Connection header received
0619 16:36:33.066814 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:33.066817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:33.066877 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:33.066960 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:36:33.067222 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.067909 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.069244 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.069876 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.069970 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.070029 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.070392 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.070467 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:36:33.070569 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.066217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32855 (local address 127.0.98.3:50599)
0619 16:36:33.066605 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.066617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:36:33.066706 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:36:33.066998 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:36:33.067000 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:36:33.067017 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:36:33.067108 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.067113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.068015 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.068018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.069130 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:36:33.069139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.070009 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.070013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:36:33.070202 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.070265 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 16:36:33.070342 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:36:33.070409 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":94,"thread_start_us":136,"threads_started":1}
I20260619 16:36:33.071157 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a27c9c33374c6092ab8db3342d3223" candidate_uuid: "f371941bb5384f44a66951c50f7c6204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0550f83249d14e25ab0c593e60fabee7" is_pre_election: true
I20260619 16:36:33.071161 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a27c9c33374c6092ab8db3342d3223" candidate_uuid: "f371941bb5384f44a66951c50f7c6204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" is_pre_election: true
I20260619 16:36:33.071420 10907 raft_consensus.cc:2454] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f371941bb5384f44a66951c50f7c6204 in term 0.
I20260619 16:36:33.071429 10773 raft_consensus.cc:2454] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f371941bb5384f44a66951c50f7c6204 in term 0.
I20260619 16:36:33.071857 10976 leader_election.cc:302] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e978e744bf54de6b8d63a5ca82c0afe, f371941bb5384f44a66951c50f7c6204; no voters:
I20260619 16:36:33.072099 11110 raft_consensus.cc:2781] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:36:33.072173 11110 raft_consensus.cc:482] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:36:33.072204 11110 raft_consensus.cc:3037] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:33.073223 11110 raft_consensus.cc:504] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.073582 11110 leader_election.cc:288] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [CANDIDATE]: Term 1 election: Requested vote from peers 7e978e744bf54de6b8d63a5ca82c0afe (127.0.98.1:40401), 0550f83249d14e25ab0c593e60fabee7 (127.0.98.2:32855)
I20260619 16:36:33.073871 10773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a27c9c33374c6092ab8db3342d3223" candidate_uuid: "f371941bb5384f44a66951c50f7c6204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e978e744bf54de6b8d63a5ca82c0afe"
I20260619 16:36:33.073958 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a27c9c33374c6092ab8db3342d3223" candidate_uuid: "f371941bb5384f44a66951c50f7c6204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0550f83249d14e25ab0c593e60fabee7"
I20260619 16:36:33.074029 10773 raft_consensus.cc:3037] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:33.074079 10907 raft_consensus.cc:3037] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:36:33.074999 10773 raft_consensus.cc:2454] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f371941bb5384f44a66951c50f7c6204 in term 1.
I20260619 16:36:33.075031 10907 raft_consensus.cc:2454] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f371941bb5384f44a66951c50f7c6204 in term 1.
I20260619 16:36:33.075266 10976 leader_election.cc:302] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e978e744bf54de6b8d63a5ca82c0afe, f371941bb5384f44a66951c50f7c6204; no voters:
I20260619 16:36:33.075440 11110 raft_consensus.cc:2781] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:36:33.075670 11110 raft_consensus.cc:686] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 1 LEADER]: Becoming Leader. State: Replica: f371941bb5384f44a66951c50f7c6204, State: Running, Role: LEADER
I20260619 16:36:33.075996 11110 consensus_queue.cc:237] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } }
I20260619 16:36:33.079071 10626 catalog_manager.cc:5495] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 reported cstate change: term changed from 0 to 1, leader changed from <none> to f371941bb5384f44a66951c50f7c6204 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f371941bb5384f44a66951c50f7c6204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f371941bb5384f44a66951c50f7c6204" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 } health_report { overall_health: UNKNOWN } } }
I20260619 16:36:33.099884 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.096411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.0.98.3:38801)
0619 16:36:33.096546 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:36:33.096550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:36:33.096562 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:36:33.096632 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:36:33.096634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:36:33.096682 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:36:33.096771 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:36:33.097014 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.097577 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.098455 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:36:33.098670 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:36:33.098780 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:36:33.099060 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:36:33.099488 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:36:33.099617 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:36:33.099664 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:36:33.099717 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:36:33.105412 10907 raft_consensus.cc:1261] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 1 FOLLOWER]: Refusing update from remote peer f371941bb5384f44a66951c50f7c6204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:33.105428 10773 raft_consensus.cc:1261] T 48a27c9c33374c6092ab8db3342d3223 P 7e978e744bf54de6b8d63a5ca82c0afe [term 1 FOLLOWER]: Refusing update from remote peer f371941bb5384f44a66951c50f7c6204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:36:33.106094 11110 consensus_queue.cc:1035] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e978e744bf54de6b8d63a5ca82c0afe" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:33.106310 11114 consensus_queue.cc:1035] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0550f83249d14e25ab0c593e60fabee7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:36:33.114740 11119 mvcc.cc:205] Tried to move back new op lower bound from 7298609123752361984 to 7298609123642220544. Current Snapshot: MvccSnapshot[applied={T|T < 7298609123752361984}]
I20260619 16:36:33.121251 10626 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:33.121456 10626 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:36:33.123584 10626 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 074a68eac28f449d9a901e5c6fbcdaa3: Sending DeleteTablet for 3 replicas of tablet 48a27c9c33374c6092ab8db3342d3223
I20260619 16:36:33.124276 11021 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a27c9c33374c6092ab8db3342d3223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48752
I20260619 16:36:33.124326 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a27c9c33374c6092ab8db3342d3223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35074
I20260619 16:36:33.124380 10753 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a27c9c33374c6092ab8db3342d3223 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:36:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59508
I20260619 16:36:33.124720 11127 tablet_replica.cc:330] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: stopping tablet replica
I20260619 16:36:33.125015 11127 raft_consensus.cc:2229] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:36:33.125180 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10690
I20260619 16:36:33.126437 11127 raft_consensus.cc:2258] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:33.127954 11127 ts_tablet_manager.cc:1905] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:36:33.128619 11129 tablet_replica.cc:330] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7: stopping tablet replica
I20260619 16:36:33.128892 11129 raft_consensus.cc:2229] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:36:33.129158 11129 raft_consensus.cc:2258] T 48a27c9c33374c6092ab8db3342d3223 P 0550f83249d14e25ab0c593e60fabee7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:36:33.130882 11127 ts_tablet_manager.cc:1918] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:36:33.130990 11127 log.cc:1196] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/48a27c9c33374c6092ab8db3342d3223
I20260619 16:36:33.131313 11127 ts_tablet_manager.cc:1939] T 48a27c9c33374c6092ab8db3342d3223 P f371941bb5384f44a66951c50f7c6204: Deleting consensus metadata
I20260619 16:36:33.132473 10613 catalog_manager.cc:4841] TS f371941bb5384f44a66951c50f7c6204 (127.0.98.3:38801): tablet 48a27c9c33374c6092ab8db3342d3223 (table test-table [id=f82890c232f74f4da95420448dc625db]) successfully deleted
W20260619 16:36:33.133901 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40401 (error 108)
I20260619 16:36:33.134697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10824
I20260619 16:36:33.134917 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.134569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40401 (local address 127.0.0.1:59512)
0619 16:36:33.134674 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.134739 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 16:36:33.135334 10614 catalog_manager.cc:4584] TS 7e978e744bf54de6b8d63a5ca82c0afe (127.0.98.1:40401): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48a27c9c33374c6092ab8db3342d3223: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40401: connect: Connection refused (error 111)
I20260619 16:36:33.141254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10958
I20260619 16:36:33.141541 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:36:33.141226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32855 (local address 127.0.0.1:35080)
0619 16:36:33.141396 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:36:33.141454 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:36:33.146858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 10596
2026-06-19T16:36:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2221 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:36:33 dist-test-slave-0f28 krb5kdc[11133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:36:33 dist-test-slave-0f28 krb5kdc[11133](info): set up 2 sockets
Jun 19 16:36:33 dist-test-slave-0f28 krb5kdc[11133](info): commencing operation
krb5kdc: starting...
W20260619 16:36:34.190403 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:36:34 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781886994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:36:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:36:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 16:36:34.352741 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFXRShK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/postgres -l logfile start
2026-06-19 16:36:36.936 UTC [11163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:36:36.936 UTC [11163] LOG: listening on IPv4 address "127.0.98.20", port 52343
2026-06-19 16:36:36.940 UTC [11163] LOG: listening on Unix socket "/tmp/.s.PGSQL.52343"
2026-06-19 16:36:36.961 UTC [11166] LOG: database system was shut down at 2026-06-19 16:36:35 UTC
2026-06-19 16:36:36.965 UTC [11163] LOG: database system is ready to accept connections
I20260619 16:36:37.962034 392 mini_postgres.cc:96] Postgres bound to 52343
2026-06-19 16:36:37.968 UTC [11174] FATAL: database "slave" does not exist
127.0.98.20:52343 - accepting connections
I20260619 16:36:37.968803 392 mini_ranger.cc:161] Starting Ranger...
I20260619 16:36:37.983469 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:36:38.522908 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:36:38.523011 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-admin
2026-06-19 16:36:38,794 [I] DB FLAVOR :POSTGRES
2026-06-19 16:36:38,794 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:36:38,794 [I] Checking connection..
2026-06-19 16:36:38,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:36:39,056 [I] Checking connection passed.
2026-06-19 16:36:39,056 [I] --------- Verifying version history table ---------
2026-06-19 16:36:39,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:36:39,305 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:36:39,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:36:39,535 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:36:39,535 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:36:39,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:36:39.784 UTC [11238] WARNING: there is no transaction in progress
2026-06-19 16:36:39,790 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:36:39,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:36:40,046 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:36:40,046 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:36:40,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:36:40,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:36:40,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0f28.c.gcp-upstream.internal','N') ;"
2026-06-19 16:36:40,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:36:41,043 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:36:41,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:36:41,272 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:36:41,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:36:41,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:36:41,735 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:36:41,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:36:42.692 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.709 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.725 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.741 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.757 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.775 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.835 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.842 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.856 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.869 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.879 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.893 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.903 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.910 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.917 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:42.924 UTC [11346] WARNING: there is no transaction in progress
2026-06-19 16:36:43,599 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:36:43,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:36:43,815 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:36:43,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:36:44,039 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:36:44,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:36:44,287 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:36:44,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:36:44,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:36:44,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0f28.c.gcp-upstream.internal';"
2026-06-19 16:36:45,016 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:36:45,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:36:45,268 [I] DB_PATCHES have already been applied
I20260619 16:36:45.274410 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFXRShK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:36:45.274472 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 16:36:45.277815 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:46569
Jun 19, 2026 4:36:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:36:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:36:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:36:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:36:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:36:46.378116 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.409s user 0.002s sys 0.007s
Jun 19, 2026 4:37:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:37:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 4:37:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:37:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:37:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:37:07.350332 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:37:07 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887027, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:37:07.438298 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:37:07.448657 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:37:08.431787 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:37:08.431882 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms
I20260619 16:37:08.431979 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52343
2026-06-19 16:37:08,614 [I] DB FLAVOR :POSTGRES
2026-06-19 16:37:08,614 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:37:08,615 [I] Checking connection
2026-06-19 16:37:08,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:37:08,870 [I] connection success
2026-06-19 16:37:08,870 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:37:08,870 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:37:08,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:37:09,094 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:37:09,094 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:37:09,094 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:37:09,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52343/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:37:09,342 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:37:09.767737 392 mini_ranger_kms.cc:310] Created kms service
I20260619 16:37:09.953606 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:37:09.996097 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:37:09 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887027, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:37:10.366758 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:37:10.366915 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:37:10.366938 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 16:37:10.370630 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11526
I20260619 16:37:10.370754 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59393
Jun 19, 2026 4:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:37:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:37:12.624259 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.186s user 0.009s sys 0.157s
I20260619 16:37:12.624404 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:37:12.624536 392 mini_ranger_kms.cc:401] 127.0.98.20:59393/kms/v1/keys
Jun 19 16:37:12 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887032, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:37:13 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887032, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:37:13 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887032, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:37:15 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:37:15.742761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42659
--ranger_config_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59393/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:37:15.852619 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:37:15.852862 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:37:15.852900 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:37:15.856837 11577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:37:15.856892 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:37:15.856911 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:37:15.856927 11577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:37:15.856942 11577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:37:15.862519 11577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59393/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42659
--ranger_config_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:37:15.863457 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:37:15.864375 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:37:15.873039 11577 server_base.cc:1034] running on GCE node
W20260619 16:37:15.873090 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:15.873090 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:15.873067 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:37:15.873909 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:37:15.875016 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:37:15.876202 11577 hybrid_clock.cc:648] HybridClock initialized: now 1781887035876181 us; error 29 us; skew 500 ppm
Jun 19 16:37:15 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:37:15.882863 11577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:37:15.883773 11577 webserver.cc:460] Webserver started at http://127.0.98.62:41535/ using document root <none> and password file <none>
I20260619 16:37:15.884303 11577 fs_manager.cc:359] Metadata directory not provided
I20260619 16:37:15.884383 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:37:15.884594 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:37:15 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:37:15.944718 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "0372d6a0b2d34aa6b8ffb932ea171b30"
format_stamp: "Formatted at 2026-06-19 16:37:15 on dist-test-slave-0f28"
server_key: "39f4f8b20cace4540862b6df34f02278"
server_key_iv: "fe54ee0434b9ca4092f1a75397c9a67a"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:15.945413 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "0372d6a0b2d34aa6b8ffb932ea171b30"
format_stamp: "Formatted at 2026-06-19 16:37:15 on dist-test-slave-0f28"
server_key: "39f4f8b20cace4540862b6df34f02278"
server_key_iv: "fe54ee0434b9ca4092f1a75397c9a67a"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:15.949574 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:37:15.968189 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:15.969698 11577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:37:15.969844 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "0372d6a0b2d34aa6b8ffb932ea171b30"
format_stamp: "Formatted at 2026-06-19 16:37:15 on dist-test-slave-0f28"
server_key: "39f4f8b20cace4540862b6df34f02278"
server_key_iv: "fe54ee0434b9ca4092f1a75397c9a67a"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:15.969990 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:37:16.000316 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:37:16.003321 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:37:16.003544 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:37:16.012033 11577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42659
I20260619 16:37:16.012089 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42659 every 8 connection(s)
I20260619 16:37:16.013006 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:37:16.016405 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:37:16.020857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 11577
I20260619 16:37:16.020993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:37:16.021271 392 external_mini_cluster.cc:1428] Setting key 13ded2982686ce7e22489cf51eda0852
I20260619 16:37:16.023347 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30: Bootstrap starting.
I20260619 16:37:16.025961 11645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30: Neither blocks nor log segments found. Creating new log.
I20260619 16:37:16.026829 11645 log.cc:824] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30: Log is configured to *not* fsync() on all Append() calls
I20260619 16:37:16.028937 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30: No bootstrap required, opened a new log
Jun 19 16:37:16 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:37:16.033428 11645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42659 } }
I20260619 16:37:16.033725 11645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:37:16.033812 11645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0372d6a0b2d34aa6b8ffb932ea171b30, State: Initialized, Role: FOLLOWER
I20260619 16:37:16.034334 11645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42659 } }
I20260619 16:37:16.034677 11645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:37:16.034780 11645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:37:16.034886 11645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:37:16.036124 11645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42659 } }
I20260619 16:37:16.036594 11645 leader_election.cc:302] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0372d6a0b2d34aa6b8ffb932ea171b30; no voters:
I20260619 16:37:16.036679 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:16.022719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37426 (local address 127.0.98.62:42659)
0619 16:37:16.023120 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:16.023144 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:16.023178 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:37:16.023834 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:16.023857 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:16.024165 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:16.024488 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:37:16.025245 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:16.026361 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:16.027322 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:16.028043 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:16.028203 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:16.031394 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:37:16.031400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:37:16.031417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:37:16.031447 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:37:16.033462 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:16.033991 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:16.033997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:16.034003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:16.034133 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:16.034445 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:16.034448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:16.034450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:16.034939 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:37:16.035178 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:16.035463 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 16:37:16.035734 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260619 16:37:16.037122 11650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:37:16.037338 11650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [term 1 LEADER]: Becoming Leader. State: Replica: 0372d6a0b2d34aa6b8ffb932ea171b30, State: Running, Role: LEADER
I20260619 16:37:16.037706 11650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42659 } }
I20260619 16:37:16.037956 11645 leader_election.cc:288] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:37:16.038913 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42659 } } }
I20260619 16:37:16.039090 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [sys.catalog]: This master's current role is: LEADER
I20260619 16:37:16.039402 11651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0372d6a0b2d34aa6b8ffb932ea171b30. Latest consensus state: current_term: 1 leader_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0372d6a0b2d34aa6b8ffb932ea171b30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42659 } } }
I20260619 16:37:16.039533 11651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [sys.catalog]: This master's current role is: LEADER
I20260619 16:37:16.039767 11645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:37:16.041843 11654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:37:16.042089 11645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:37:16.047690 11654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:37:16.054421 11654 catalog_manager.cc:1347] Generated new cluster ID: 363464f64dc54496b5026c535e80e963
I20260619 16:37:16.054482 11654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:37:16.071668 11654 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:37:16.072985 11654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:37:16.079608 11654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30: Generated new TSK 0
I20260619 16:37:16.080457 11654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:37:16.745150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59393/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42659
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:37:16.890666 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:37:16.891073 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:37:16.891178 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:37:16.896504 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:37:16.896559 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:37:16.896646 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:37:16.902309 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59393/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:37:16.903327 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:37:16.904320 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:37:16.913733 11697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:16.913767 11700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:16.913767 11698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:37:16.914173 11690 server_base.cc:1034] running on GCE node
I20260619 16:37:16.914567 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:37:16.915166 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:37:16.916354 11690 hybrid_clock.cc:648] HybridClock initialized: now 1781887036916337 us; error 36 us; skew 500 ppm
Jun 19 16:37:16 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:37:16.919679 11690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:37:16.920493 11690 webserver.cc:460] Webserver started at http://127.0.98.1:39453/ using document root <none> and password file <none>
I20260619 16:37:16.921098 11690 fs_manager.cc:359] Metadata directory not provided
I20260619 16:37:16.921182 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:37:16.921475 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:37:16 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:37:16.942366 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "66779e92cf6a4fb19f478a194f873484"
format_stamp: "Formatted at 2026-06-19 16:37:16 on dist-test-slave-0f28"
server_key: "31889cd92d6bc3e8a487cf9e4c1c112e"
server_key_iv: "0a883cc397016db564c3571059279980"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:16.942943 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66779e92cf6a4fb19f478a194f873484"
format_stamp: "Formatted at 2026-06-19 16:37:16 on dist-test-slave-0f28"
server_key: "31889cd92d6bc3e8a487cf9e4c1c112e"
server_key_iv: "0a883cc397016db564c3571059279980"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:16.946807 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 16:37:16.962913 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:16.968770 11690 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.004s
I20260619 16:37:16.968927 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "66779e92cf6a4fb19f478a194f873484"
format_stamp: "Formatted at 2026-06-19 16:37:16 on dist-test-slave-0f28"
server_key: "31889cd92d6bc3e8a487cf9e4c1c112e"
server_key_iv: "0a883cc397016db564c3571059279980"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:16.969061 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:37:16.997080 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:37:17.000695 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:37:17.000931 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:37:17.001641 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:37:17.002656 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:37:17.002764 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.002893 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:37:17.002987 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.024315 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46355
I20260619 16:37:17.030052 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:37:17.034647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 11690
I20260619 16:37:17.034780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:37:17.035101 392 external_mini_cluster.cc:1428] Setting key 1ba2b6f30741e9c28eade5b466363b04
I20260619 16:37:17.032215 11822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46355 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:37:17.057274 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.034915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42659 (local address 127.0.98.1:35757)
0619 16:37:17.036484 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.036539 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.037571 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.041478 (+ 3907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.041489 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.041888 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:37:17.042619 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.042642 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.045438 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.045444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.046694 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.046703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.047332 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.047338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.047520 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.048257 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:37:17.048278 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:37:17.050085 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:37:17.054131 (+ 4046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:37:17.054137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:37:17.054153 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:37:17.054480 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:37:17.054780 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:37:17.054783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:37:17.054785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:37:17.054946 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:37:17.055422 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:37:17.055429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:37:17.055731 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.055911 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.056222 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":130,"threads_started":1}
I20260619 16:37:17.058650 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.034388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35757 (local address 127.0.98.62:42659)
0619 16:37:17.040981 (+ 6593us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.040991 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.041009 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:37:17.041055 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.041061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.041117 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.042816 (+ 1699us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:37:17.042948 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.043907 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.046828 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.047173 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.047290 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.050291 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:37:17.050295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:37:17.050298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:37:17.050323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:37:17.054002 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.054600 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.054603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.054605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.054685 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.055061 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.055064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.055066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.055301 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:37:17.058283 (+ 2982us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.058362 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.058486 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6494,"thread_start_us":109,"threads_started":1}
I20260619 16:37:17.059939 11823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42659
I20260619 16:37:17.060189 11823 heartbeater.cc:464] Registering TS with master...
I20260619 16:37:17.060731 11823 heartbeater.cc:511] Master 127.0.98.62:42659 requested a full tablet report, sending...
I20260619 16:37:17.062530 11609 ts_manager.cc:194] Registered new tserver with Master: 66779e92cf6a4fb19f478a194f873484 (127.0.98.1:46355)
I20260619 16:37:17.064195 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:37:17.077253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59393/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42659
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:37:17.237942 11829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:37:17.238225 11829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:37:17.238291 11829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:37:17.242888 11829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:37:17.242964 11829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:37:17.243096 11829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:37:17.248261 11829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59393/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:37:17.249411 11829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:37:17.250556 11829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:37:17.261494 11835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:17.261494 11838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:17.261665 11836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:37:17.262275 11829 server_base.cc:1034] running on GCE node
I20260619 16:37:17.262686 11829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:37:17.263300 11829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:37:17.264505 11829 hybrid_clock.cc:648] HybridClock initialized: now 1781887037264485 us; error 36 us; skew 500 ppm
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:37:17.267896 11829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:37:17.268960 11829 webserver.cc:460] Webserver started at http://127.0.98.2:44481/ using document root <none> and password file <none>
I20260619 16:37:17.269722 11829 fs_manager.cc:359] Metadata directory not provided
I20260619 16:37:17.269832 11829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:37:17.270099 11829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:37:17.287631 11829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "5404c853530044dfabbaa1d64872bbcd"
format_stamp: "Formatted at 2026-06-19 16:37:17 on dist-test-slave-0f28"
server_key: "4cdcfa02876120fb702a84d853b2ad8d"
server_key_iv: "23bfeddc897a3eff18441faff9cc5f7b"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:17.288371 11829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5404c853530044dfabbaa1d64872bbcd"
format_stamp: "Formatted at 2026-06-19 16:37:17 on dist-test-slave-0f28"
server_key: "4cdcfa02876120fb702a84d853b2ad8d"
server_key_iv: "23bfeddc897a3eff18441faff9cc5f7b"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:17.294472 11829 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260619 16:37:17.316567 11845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.318234 11829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:37:17.318382 11829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "5404c853530044dfabbaa1d64872bbcd"
format_stamp: "Formatted at 2026-06-19 16:37:17 on dist-test-slave-0f28"
server_key: "4cdcfa02876120fb702a84d853b2ad8d"
server_key_iv: "23bfeddc897a3eff18441faff9cc5f7b"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:17.318532 11829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:37:17.371450 11829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:37:17.374989 11829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:37:17.375195 11829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:37:17.375825 11829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:37:17.376690 11829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:37:17.376767 11829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.376839 11829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:37:17.376875 11829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.388091 11829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35531
I20260619 16:37:17.388130 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35531 every 8 connection(s)
I20260619 16:37:17.388993 11829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:37:17.392985 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 11829
I20260619 16:37:17.393136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:37:17.393537 392 external_mini_cluster.cc:1428] Setting key 66f6d028ad4b0ad15a00aef2799887a7
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:37:17.409713 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.390942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39545 (local address 127.0.98.62:42659)
0619 16:37:17.391104 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.391109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.391912 (+ 803us) server_negotiation.cc:407] Connection header received
0619 16:37:17.393400 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.393402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.393448 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.393545 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:37:17.394751 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.395401 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.397679 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.398100 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.398222 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.401808 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:37:17.401814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:37:17.401819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:37:17.401856 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:37:17.407101 (+ 5245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.407796 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.407800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.407801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.407896 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.408305 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.408309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.408310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.408552 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:37:17.408632 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.409400 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.409540 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:37:17.411671 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.391295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42659 (local address 127.0.98.2:39545)
0619 16:37:17.391741 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.391798 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.392611 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.393743 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.393750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.394067 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:37:17.394612 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.394626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.395543 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.395548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.397534 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.397549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.398681 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.398687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.398914 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.399683 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:37:17.399707 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:37:17.401584 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:37:17.407242 (+ 5658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:37:17.407249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:37:17.407267 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:37:17.407663 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:37:17.408013 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:37:17.408016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:37:17.408018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:37:17.408193 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:37:17.408719 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:37:17.408726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:37:17.409069 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.410341 (+ 1272us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.410673 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":135,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:37:17.413941 11959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42659
I20260619 16:37:17.414186 11959 heartbeater.cc:464] Registering TS with master...
I20260619 16:37:17.414779 11959 heartbeater.cc:511] Master 127.0.98.62:42659 requested a full tablet report, sending...
I20260619 16:37:17.415923 11609 ts_manager.cc:194] Registered new tserver with Master: 5404c853530044dfabbaa1d64872bbcd (127.0.98.2:35531)
I20260619 16:37:17.416661 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:37:17.434950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59393/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42659
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 16:37:17.593924 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:37:17.594331 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:37:17.594429 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:37:17.600325 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:37:17.600471 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:37:17.600592 11964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:37:17.607968 11964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59393/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:37:17.609603 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:37:17.611032 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:37:17.624141 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:17.624178 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:37:17.624792 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:37:17.625952 11964 server_base.cc:1034] running on GCE node
I20260619 16:37:17.626492 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:37:17.627192 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:37:17.628371 11964 hybrid_clock.cc:648] HybridClock initialized: now 1781887037628310 us; error 73 us; skew 500 ppm
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:37:17.631852 11964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:37:17.632966 11964 webserver.cc:460] Webserver started at http://127.0.98.3:41179/ using document root <none> and password file <none>
I20260619 16:37:17.633782 11964 fs_manager.cc:359] Metadata directory not provided
I20260619 16:37:17.633929 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:37:17.634199 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:37:17.665110 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "166a1dd58457441ea977b42adb681b45"
format_stamp: "Formatted at 2026-06-19 16:37:17 on dist-test-slave-0f28"
server_key: "7dabe38490ef1acc5f20f93b08348871"
server_key_iv: "c63aaffa7714ff10e1aec1c5bf55d7cc"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:17.665951 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "166a1dd58457441ea977b42adb681b45"
format_stamp: "Formatted at 2026-06-19 16:37:17 on dist-test-slave-0f28"
server_key: "7dabe38490ef1acc5f20f93b08348871"
server_key_iv: "c63aaffa7714ff10e1aec1c5bf55d7cc"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:17.670158 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:37:17.685920 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.687541 11964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:37:17.687750 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "166a1dd58457441ea977b42adb681b45"
format_stamp: "Formatted at 2026-06-19 16:37:17 on dist-test-slave-0f28"
server_key: "7dabe38490ef1acc5f20f93b08348871"
server_key_iv: "c63aaffa7714ff10e1aec1c5bf55d7cc"
server_key_version: "kuduclusterkey@0"
I20260619 16:37:17.687954 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:37:17.706005 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:37:17.709779 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:37:17.710064 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:37:17.710772 11964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:37:17.711838 11964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:37:17.711913 11964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.712021 11964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:37:17.712080 11964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:37:17.724243 11964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36253
I20260619 16:37:17.724503 12092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36253 every 8 connection(s)
I20260619 16:37:17.725297 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:37:17.730561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 11964
I20260619 16:37:17.730685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:37:17.730971 392 external_mini_cluster.cc:1428] Setting key 5781c9aebac530e6750ad311221ea25b
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:37:17.739634 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.727235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59533 (local address 127.0.98.62:42659)
0619 16:37:17.727365 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.727370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.728274 (+ 904us) server_negotiation.cc:407] Connection header received
0619 16:37:17.729245 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.729249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.729306 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.729441 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:37:17.730609 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.731391 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.733269 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.733583 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.733697 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.735997 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:37:17.736001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:37:17.736004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:37:17.736035 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:37:17.737503 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.737999 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.738003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.738005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.738086 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.738427 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.738430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.738431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.738608 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:37:17.738707 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.739354 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.739501 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:37:17.741065 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.727673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42659 (local address 127.0.98.3:59533)
0619 16:37:17.728123 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.728172 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.729037 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.729611 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.729617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.729884 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:37:17.730423 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.730444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.731777 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.731780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.733143 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.733154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.733730 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.733735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.733913 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.734517 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:37:17.734532 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:37:17.735839 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:37:17.737616 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:37:17.737622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:37:17.737636 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:37:17.737893 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:37:17.738181 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:37:17.738184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:37:17.738186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:37:17.738322 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:37:17.738720 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:37:17.738727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:37:17.739025 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.739869 (+ 844us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.740176 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":148,"threads_started":1}
I20260619 16:37:17.742633 12093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42659
I20260619 16:37:17.742858 12093 heartbeater.cc:464] Registering TS with master...
I20260619 16:37:17.743320 12093 heartbeater.cc:511] Master 127.0.98.62:42659 requested a full tablet report, sending...
I20260619 16:37:17.744455 11609 ts_manager.cc:194] Registered new tserver with Master: 166a1dd58457441ea977b42adb681b45 (127.0.98.3:36253)
I20260619 16:37:17.744853 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:37:17.745151 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59533
I20260619 16:37:17.755105 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.747181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.0.98.62:42659)
0619 16:37:17.747289 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.747293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.747766 (+ 473us) server_negotiation.cc:407] Connection header received
0619 16:37:17.747912 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.747915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.747955 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.748046 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:37:17.748752 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.749257 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.750152 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.750387 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.750498 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.751884 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:37:17.751888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:37:17.751891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:37:17.751915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:37:17.753427 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.753847 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.753850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.753852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.753935 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:37:17.754244 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:37:17.754249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:37:17.754251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:37:17.754426 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:37:17.754563 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.754814 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.754936 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:37:17.758221 11609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:37:17.761261 11609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:37:17.779089 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.770054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46355 (local address 127.0.0.1:37602)
0619 16:37:17.771963 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.772009 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.772204 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.772962 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.772968 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.772988 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:37:17.773144 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.773151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.775363 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.775367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.776752 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.776761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.778542 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.778546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.778652 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.778718 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.778815 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.778933 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1746,"mutex_wait_us":3,"thread_start_us":538,"threads_started":1}
I20260619 16:37:17.779841 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.771289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.0.98.1:46355)
0619 16:37:17.771705 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.771715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.772096 (+ 381us) server_negotiation.cc:407] Connection header received
0619 16:37:17.772402 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.772409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.772620 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.772750 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:37:17.773306 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.774553 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.776888 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.777732 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.779394 (+ 1662us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.779542 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.779591 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.779691 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":102,"threads_started":1}
I20260619 16:37:17.783196 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.771380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36253 (local address 127.0.0.1:42312)
0619 16:37:17.772769 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.772782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.772868 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.775292 (+ 2424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.775295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.775315 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:37:17.776842 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.776850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.779750 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.779754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.781604 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.781616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.782811 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.782815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.782918 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.782944 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.783016 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.783076 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":1342,"thread_start_us":71,"threads_started":1}
I20260619 16:37:17.782899 11756 tablet_service.cc:1467] Processing CreateTablet for tablet 160ca88763fd4fdf990916e190678eb1 (DEFAULT_TABLE table=test-table [id=c1619f5b894f485eafa0a40247f9623e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:37:17.784036 11756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160ca88763fd4fdf990916e190678eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:37:17.786285 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.771666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.0.1:41030)
0619 16:37:17.773037 (+ 1371us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.773049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.773135 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.774808 (+ 1673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.774811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.774830 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:37:17.774980 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.774989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.776945 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.776947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.778067 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.778076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.784849 (+ 6773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.784852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.785965 (+ 1113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.786018 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.786095 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.786161 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"thread_start_us":78,"threads_started":1}
I20260619 16:37:17.786988 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.772203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.0.98.3:36253)
0619 16:37:17.773563 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.773573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.773913 (+ 340us) server_negotiation.cc:407] Connection header received
0619 16:37:17.773990 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.773995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.774162 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.775148 (+ 986us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:37:17.778197 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.779628 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.781758 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.782518 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.782623 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.782750 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.786741 (+ 3991us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.786846 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":141,"threads_started":1}
I20260619 16:37:17.787463 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.771840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41030 (local address 127.0.98.2:35531)
0619 16:37:17.773795 (+ 1955us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.773805 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.773825 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:37:17.774345 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.774350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.774500 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.774639 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:37:17.775520 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.776628 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.783486 (+ 6858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.784567 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.784665 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.784785 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.787209 (+ 2424us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.787310 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1860,"thread_start_us":124,"threads_started":1}
I20260619 16:37:17.790593 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 160ca88763fd4fdf990916e190678eb1 (DEFAULT_TABLE table=test-table [id=c1619f5b894f485eafa0a40247f9623e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:37:17.791739 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160ca88763fd4fdf990916e190678eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:37:17.798131 12111 tablet_bootstrap.cc:492] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: Bootstrap starting.
I20260619 16:37:17.800554 12111 tablet_bootstrap.cc:654] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: Neither blocks nor log segments found. Creating new log.
I20260619 16:37:17.800591 11893 tablet_service.cc:1467] Processing CreateTablet for tablet 160ca88763fd4fdf990916e190678eb1 (DEFAULT_TABLE table=test-table [id=c1619f5b894f485eafa0a40247f9623e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:37:17.801450 12111 log.cc:824] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: Log is configured to *not* fsync() on all Append() calls
I20260619 16:37:17.801687 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 160ca88763fd4fdf990916e190678eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:37:17.803431 12111 tablet_bootstrap.cc:492] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: No bootstrap required, opened a new log
I20260619 16:37:17.803689 12111 ts_tablet_manager.cc:1397] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:37:17.808769 12111 raft_consensus.cc:348] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.809157 12111 raft_consensus.cc:374] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:37:17.809247 12111 raft_consensus.cc:729] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66779e92cf6a4fb19f478a194f873484, State: Initialized, Role: FOLLOWER
I20260619 16:37:17.809669 12112 tablet_bootstrap.cc:492] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Bootstrap starting.
I20260619 16:37:17.809710 12114 tablet_bootstrap.cc:492] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Bootstrap starting.
I20260619 16:37:17.809783 12111 consensus_queue.cc:260] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.810834 12111 ts_tablet_manager.cc:1428] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:37:17.811213 11823 heartbeater.cc:503] Master 127.0.98.62:42659 was elected leader, sending a full tablet report...
I20260619 16:37:17.812356 12114 tablet_bootstrap.cc:654] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Neither blocks nor log segments found. Creating new log.
I20260619 16:37:17.813233 12114 log.cc:824] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:37:17.815405 12114 tablet_bootstrap.cc:492] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: No bootstrap required, opened a new log
I20260619 16:37:17.815640 12114 ts_tablet_manager.cc:1397] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260619 16:37:17.816253 12112 tablet_bootstrap.cc:654] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Neither blocks nor log segments found. Creating new log.
I20260619 16:37:17.817078 12112 log.cc:824] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Log is configured to *not* fsync() on all Append() calls
I20260619 16:37:17.819666 12114 raft_consensus.cc:348] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.819921 12114 raft_consensus.cc:374] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:37:17.820011 12114 raft_consensus.cc:729] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5404c853530044dfabbaa1d64872bbcd, State: Initialized, Role: FOLLOWER
I20260619 16:37:17.820473 12114 consensus_queue.cc:260] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.821429 12114 ts_tablet_manager.cc:1428] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:37:17.821815 11959 heartbeater.cc:503] Master 127.0.98.62:42659 was elected leader, sending a full tablet report...
I20260619 16:37:17.823132 12112 tablet_bootstrap.cc:492] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: No bootstrap required, opened a new log
I20260619 16:37:17.823343 12112 ts_tablet_manager.cc:1397] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260619 16:37:17.827131 12112 raft_consensus.cc:348] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.827387 12112 raft_consensus.cc:374] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:37:17.827481 12112 raft_consensus.cc:729] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166a1dd58457441ea977b42adb681b45, State: Initialized, Role: FOLLOWER
I20260619 16:37:17.827921 12112 consensus_queue.cc:260] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.828547 12093 heartbeater.cc:503] Master 127.0.98.62:42659 was elected leader, sending a full tablet report...
I20260619 16:37:17.828896 12112 ts_tablet_manager.cc:1428] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
Jun 19 16:37:17 dist-test-slave-0f28 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 16:37:17.890188 11960 tablet.cc:2369] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:37:17.953293 12115 raft_consensus.cc:482] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:37:17.953583 12115 raft_consensus.cc:504] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.955008 12115 leader_election.cc:288] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 166a1dd58457441ea977b42adb681b45 (127.0.98.3:36253), 5404c853530044dfabbaa1d64872bbcd (127.0.98.2:35531)
I20260619 16:37:17.962119 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.956247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43287 (local address 127.0.98.3:36253)
0619 16:37:17.956544 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.956549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.956937 (+ 388us) server_negotiation.cc:407] Connection header received
0619 16:37:17.957153 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.957157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.957213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.957277 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:37:17.957757 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.958546 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.959708 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.960296 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.960395 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.960466 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.961920 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.961997 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260619 16:37:17.962121 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.955402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36253 (local address 127.0.98.1:43287)
0619 16:37:17.956843 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.956859 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.957022 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.957452 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.957455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.957486 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:37:17.957623 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.957631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.958718 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.958722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.959576 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.959585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.961624 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.961629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.961744 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.961782 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.961874 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.961947 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":127,"threads_started":1}
I20260619 16:37:17.962842 12032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160ca88763fd4fdf990916e190678eb1" candidate_uuid: "66779e92cf6a4fb19f478a194f873484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166a1dd58457441ea977b42adb681b45" is_pre_election: true
I20260619 16:37:17.963174 12032 raft_consensus.cc:2454] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66779e92cf6a4fb19f478a194f873484 in term 0.
I20260619 16:37:17.963721 11710 leader_election.cc:302] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166a1dd58457441ea977b42adb681b45, 66779e92cf6a4fb19f478a194f873484; no voters:
I20260619 16:37:17.964020 12115 raft_consensus.cc:2781] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:37:17.964093 12115 raft_consensus.cc:482] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:37:17.964133 12115 raft_consensus.cc:3037] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:37:17.965488 12115 raft_consensus.cc:504] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.965824 12115 leader_election.cc:288] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [CANDIDATE]: Term 1 election: Requested vote from peers 166a1dd58457441ea977b42adb681b45 (127.0.98.3:36253), 5404c853530044dfabbaa1d64872bbcd (127.0.98.2:35531)
I20260619 16:37:17.966447 12032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160ca88763fd4fdf990916e190678eb1" candidate_uuid: "66779e92cf6a4fb19f478a194f873484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166a1dd58457441ea977b42adb681b45"
I20260619 16:37:17.966559 12032 raft_consensus.cc:3037] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:37:17.967808 12032 raft_consensus.cc:2454] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66779e92cf6a4fb19f478a194f873484 in term 1.
I20260619 16:37:17.968179 11710 leader_election.cc:302] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166a1dd58457441ea977b42adb681b45, 66779e92cf6a4fb19f478a194f873484; no voters:
I20260619 16:37:17.968438 12115 raft_consensus.cc:2781] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:37:17.968643 12115 raft_consensus.cc:686] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [term 1 LEADER]: Becoming Leader. State: Replica: 66779e92cf6a4fb19f478a194f873484, State: Running, Role: LEADER
I20260619 16:37:17.968981 12115 consensus_queue.cc:237] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } }
I20260619 16:37:17.971285 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.956396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36515 (local address 127.0.98.2:35531)
0619 16:37:17.956644 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:17.956650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:17.959951 (+ 3301us) server_negotiation.cc:407] Connection header received
0619 16:37:17.960122 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:17.960125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:17.960172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:17.960218 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:37:17.960619 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.961365 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.969987 (+ 8622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.970651 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.970784 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.970850 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:17.971071 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:37:17.971160 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260619 16:37:17.971829 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:17.955809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35531 (local address 127.0.98.1:36515)
0619 16:37:17.959863 (+ 4054us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:17.959876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:37:17.960015 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:37:17.960367 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:37:17.960370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:37:17.960385 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:37:17.960487 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.960495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.961510 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.961513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.969841 (+ 8328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:37:17.969855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:37:17.970790 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:37:17.970794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:37:17.970912 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:17.970939 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 16:37:17.971586 (+ 647us) client_negotiation.cc:240] Negotiation successful
0619 16:37:17.971673 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3993,"thread_start_us":90,"threads_started":1}
I20260619 16:37:17.972536 11912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160ca88763fd4fdf990916e190678eb1" candidate_uuid: "66779e92cf6a4fb19f478a194f873484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5404c853530044dfabbaa1d64872bbcd"
I20260619 16:37:17.972550 11913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "160ca88763fd4fdf990916e190678eb1" candidate_uuid: "66779e92cf6a4fb19f478a194f873484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5404c853530044dfabbaa1d64872bbcd" is_pre_election: true
I20260619 16:37:17.972770 11912 raft_consensus.cc:3037] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:37:17.974201 11912 raft_consensus.cc:2454] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66779e92cf6a4fb19f478a194f873484 in term 1.
W20260619 16:37:17.976604 12094 tablet.cc:2369] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:37:17.977097 11609 catalog_manager.cc:5495] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 reported cstate change: term changed from 0 to 1, leader changed from <none> to 66779e92cf6a4fb19f478a194f873484 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "66779e92cf6a4fb19f478a194f873484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66779e92cf6a4fb19f478a194f873484" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 } health_report { overall_health: UNKNOWN } } }
I20260619 16:37:18.040653 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:18.033445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.0.98.1:46355)
0619 16:37:18.033604 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 16:37:18.033609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:37:18.033627 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:37:18.033706 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:37:18.033709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:37:18.033790 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:37:18.033906 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:37:18.034319 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:18.035155 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:18.036484 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:37:18.036843 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:37:18.036951 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:37:18.039332 (+ 2381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:37:18.039825 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:37:18.039943 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:37:18.040454 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 16:37:18.040515 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 16:37:18.044087 11824 tablet.cc:2369] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:37:18.049842 11912 raft_consensus.cc:1261] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 1 FOLLOWER]: Refusing update from remote peer 66779e92cf6a4fb19f478a194f873484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:37:18.050930 12032 raft_consensus.cc:1261] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 1 FOLLOWER]: Refusing update from remote peer 66779e92cf6a4fb19f478a194f873484: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:37:18.051728 12124 consensus_queue.cc:1035] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5404c853530044dfabbaa1d64872bbcd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:37:18.052079 12124 consensus_queue.cc:1035] T 160ca88763fd4fdf990916e190678eb1 P 66779e92cf6a4fb19f478a194f873484 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166a1dd58457441ea977b42adb681b45" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:37:18.068290 12127 mvcc.cc:205] Tried to move back new op lower bound from 7298609307845124096 to 7298609307524673536. Current Snapshot: MvccSnapshot[applied={T|T < 7298609307845124096}]
I20260619 16:37:18.071184 12130 mvcc.cc:205] Tried to move back new op lower bound from 7298609307845124096 to 7298609307524673536. Current Snapshot: MvccSnapshot[applied={T|T < 7298609307845124096}]
I20260619 16:37:18.076690 12128 mvcc.cc:205] Tried to move back new op lower bound from 7298609307845124096 to 7298609307524673536. Current Snapshot: MvccSnapshot[applied={T|T < 7298609307845124096}]
I20260619 16:37:18.084343 11609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:37:18.084512 11609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:37:18.089033 11609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0372d6a0b2d34aa6b8ffb932ea171b30: Sending DeleteTablet for 3 replicas of tablet 160ca88763fd4fdf990916e190678eb1
I20260619 16:37:18.089803 11756 tablet_service.cc:1514] Processing DeleteTablet for tablet 160ca88763fd4fdf990916e190678eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:37:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37602
I20260619 16:37:18.090668 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet 160ca88763fd4fdf990916e190678eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:37:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42312
I20260619 16:37:18.091588 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet 160ca88763fd4fdf990916e190678eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:37:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41030
I20260619 16:37:18.092089 12138 tablet_replica.cc:330] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: stopping tablet replica
I20260619 16:37:18.092744 12137 tablet_replica.cc:330] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: stopping tablet replica
I20260619 16:37:18.093109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 11690
I20260619 16:37:18.093171 12137 raft_consensus.cc:2229] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:37:18.097383 12138 raft_consensus.cc:2229] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:37:18.097751 12138 raft_consensus.cc:2258] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:37:18.100135 12137 raft_consensus.cc:2258] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:37:18.100695 12138 ts_tablet_manager.cc:1905] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:37:18.101682 12137 ts_tablet_manager.cc:1905] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:37:18.104482 12138 ts_tablet_manager.cc:1918] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:37:18.104647 12138 log.cc:1196] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/160ca88763fd4fdf990916e190678eb1
I20260619 16:37:18.104895 12137 ts_tablet_manager.cc:1918] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:37:18.104992 12137 log.cc:1196] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/160ca88763fd4fdf990916e190678eb1
I20260619 16:37:18.105208 12138 ts_tablet_manager.cc:1939] T 160ca88763fd4fdf990916e190678eb1 P 5404c853530044dfabbaa1d64872bbcd: Deleting consensus metadata
I20260619 16:37:18.105376 12137 ts_tablet_manager.cc:1939] T 160ca88763fd4fdf990916e190678eb1 P 166a1dd58457441ea977b42adb681b45: Deleting consensus metadata
I20260619 16:37:18.106623 11595 catalog_manager.cc:4841] TS 5404c853530044dfabbaa1d64872bbcd (127.0.98.2:35531): tablet 160ca88763fd4fdf990916e190678eb1 (table test-table [id=c1619f5b894f485eafa0a40247f9623e]) successfully deleted
I20260619 16:37:18.106858 11595 catalog_manager.cc:4841] TS 166a1dd58457441ea977b42adb681b45 (127.0.98.3:36253): tablet 160ca88763fd4fdf990916e190678eb1 (table test-table [id=c1619f5b894f485eafa0a40247f9623e]) successfully deleted
W20260619 16:37:18.106961 11596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46355 (error 108)
I20260619 16:37:18.108014 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:37:18.107681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46355 (local address 127.0.0.1:37620)
0619 16:37:18.107864 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:37:18.107929 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 16:37:18.108237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 11829
W20260619 16:37:18.109849 11596 catalog_manager.cc:4584] TS 66779e92cf6a4fb19f478a194f873484 (127.0.98.1:46355): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 160ca88763fd4fdf990916e190678eb1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46355: connect: Connection refused (error 111)
I20260619 16:37:18.116151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 11964
I20260619 16:37:18.124898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 11577
I20260619 16:37:18.136575 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:37:18.471113 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:37:18.471271 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:37:18.815889 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:37:18.816 UTC [11163] LOG: received smart shutdown request
2026-06-19 16:37:18.819 UTC [11163] LOG: background worker "logical replication launcher" (PID 11172) exited with exit code 1
2026-06-19 16:37:18.819 UTC [11167] LOG: shutting down
2026-06-19 16:37:18.998 UTC [11163] LOG: database system is shut down
2026-06-19T16:37:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45973 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:37:19 dist-test-slave-0f28 krb5kdc[12148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:37:19 dist-test-slave-0f28 krb5kdc[12148](info): set up 2 sockets
Jun 19 16:37:19 dist-test-slave-0f28 krb5kdc[12148](info): commencing operation
krb5kdc: starting...
W20260619 16:37:20.177536 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:37:20 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:37:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 16:37:20.343811 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFXRShK/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/postgres -l logfile start
2026-06-19 16:37:22.929 UTC [12178] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 16:37:22.929 UTC [12178] LOG: listening on IPv4 address "127.0.98.20", port 51383
2026-06-19 16:37:22.933 UTC [12178] LOG: listening on Unix socket "/tmp/.s.PGSQL.51383"
2026-06-19 16:37:22.967 UTC [12181] LOG: database system was shut down at 2026-06-19 16:37:21 UTC
2026-06-19 16:37:22.971 UTC [12178] LOG: database system is ready to accept connections
I20260619 16:37:23.913211 392 mini_postgres.cc:96] Postgres bound to 51383
2026-06-19 16:37:23.919 UTC [12189] FATAL: database "slave" does not exist
127.0.98.20:51383 - accepting connections
I20260619 16:37:23.920349 392 mini_ranger.cc:161] Starting Ranger...
I20260619 16:37:23.936055 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 16:37:24.474088 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 16:37:24.474164 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-admin
2026-06-19 16:37:24,764 [I] DB FLAVOR :POSTGRES
2026-06-19 16:37:24,765 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:37:24,765 [I] Checking connection..
2026-06-19 16:37:24,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 16:37:25,014 [I] Checking connection passed.
2026-06-19 16:37:25,014 [I] --------- Verifying version history table ---------
2026-06-19 16:37:25,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:37:25,261 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:37:25,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:37:25,491 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 16:37:25,492 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 16:37:25,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 16:37:25.746 UTC [12253] WARNING: there is no transaction in progress
2026-06-19 16:37:25,752 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 16:37:25,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 16:37:25,997 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 16:37:25,997 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:37:25,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 16:37:26,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 16:37:26,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0f28.c.gcp-upstream.internal','N') ;"
2026-06-19 16:37:26,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:37:26,966 [I] Table x_portal_user does not exist in database ranger
2026-06-19 16:37:26,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:37:27,214 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 16:37:27,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:37:27,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:37:27,708 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 16:37:27,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 16:37:28.703 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.719 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.736 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.752 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.769 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.790 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.847 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.854 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.870 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.881 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.892 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.906 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.916 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.923 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.930 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:28.937 UTC [12361] WARNING: there is no transaction in progress
2026-06-19 16:37:29,629 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 16:37:29,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 16:37:29,876 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 16:37:29,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 16:37:30,139 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 16:37:30,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 16:37:30,366 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 16:37:30,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 16:37:30,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 16:37:30,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0f28.c.gcp-upstream.internal';"
2026-06-19 16:37:31,054 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 16:37:31,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 16:37:31,295 [I] DB_PATCHES have already been applied
I20260619 16:37:31.301461 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFXRShK/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 16:37:31.301525 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 16:37:31.304806 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48241
Jun 19, 2026 4:37:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:37:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:37:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:37:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:37:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:37:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:37:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 16:37:32.449664 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.529s user 0.000s sys 0.010s
Jun 19, 2026 4:37:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:37:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:37:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:37:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:37:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 16:37:52.804663 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 16:37:52 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887072, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 16:37:52.890463 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 16:37:52.899670 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 16:37:53.830029 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 16:37:53.830178 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms
I20260619 16:37:53.830202 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51383
2026-06-19 16:37:54,002 [I] DB FLAVOR :POSTGRES
2026-06-19 16:37:54,002 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 16:37:54,002 [I] Checking connection
2026-06-19 16:37:54,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 16:37:54,225 [I] connection success
2026-06-19 16:37:54,225 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 16:37:54,225 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 16:37:54,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 16:37:54,463 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 16:37:54,463 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 16:37:54,463 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 16:37:54,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 16:37:54,743 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 16:37:55.206810 392 mini_ranger_kms.cc:310] Created kms service
I20260619 16:37:55.373395 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 16:37:55.419258 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 16:37:55 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887072, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:37:55.806730 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 16:37:55.806871 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFXRShK/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFXRShK/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 16:37:55.806895 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 16:37:55.810417 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12541
I20260619 16:37:55.810497 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56763
Jun 19, 2026 4:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:37:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:37:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:37:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 16:37:58.080073 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.190s user 0.024s sys 0.132s
I20260619 16:37:58.080209 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 16:37:58.080274 392 mini_ranger_kms.cc:401] 127.0.98.20:56763/kms/v1/keys
Jun 19 16:37:58 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887078, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 16:37:58 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887078, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:37:58 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887078, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 16:38:01 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:38:01.052860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33177
--ranger_config_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:01.163348 12592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:01.163677 12592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:01.163754 12592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:01.167804 12592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:01.167893 12592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:01.167932 12592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:01.167990 12592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:01.168038 12592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:01.173666 12592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33177
--ranger_config_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:01.174742 12592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:01.175781 12592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:01.184372 12597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:01.184384 12598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:01.184389 12600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:01.184609 12592 server_base.cc:1034] running on GCE node
I20260619 16:38:01.185236 12592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:01.186412 12592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:01.187611 12592 hybrid_clock.cc:648] HybridClock initialized: now 1781887081187595 us; error 35 us; skew 500 ppm
Jun 19 16:38:01 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:01.194366 12592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:01.195441 12592 webserver.cc:460] Webserver started at http://127.0.98.62:43753/ using document root <none> and password file <none>
I20260619 16:38:01.195995 12592 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:01.196094 12592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:01.196307 12592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:38:01 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:01.249919 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "ba89934c0f6649658f0328cf4bc949ed"
format_stamp: "Formatted at 2026-06-19 16:38:01 on dist-test-slave-0f28"
server_key: "61719f0001f9776b12ab039c2bbffb8d"
server_key_iv: "56fc77d25371d0d30d977074ca8ae6e5"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:01.250536 12592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba89934c0f6649658f0328cf4bc949ed"
format_stamp: "Formatted at 2026-06-19 16:38:01 on dist-test-slave-0f28"
server_key: "61719f0001f9776b12ab039c2bbffb8d"
server_key_iv: "56fc77d25371d0d30d977074ca8ae6e5"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:01.254367 12592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:38:01.271864 12607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:01.273278 12592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:38:01.273455 12592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "ba89934c0f6649658f0328cf4bc949ed"
format_stamp: "Formatted at 2026-06-19 16:38:01 on dist-test-slave-0f28"
server_key: "61719f0001f9776b12ab039c2bbffb8d"
server_key_iv: "56fc77d25371d0d30d977074ca8ae6e5"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:01.273600 12592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:01.291821 12592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:01.294370 12592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:01.294579 12592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:01.301877 12592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33177
I20260619 16:38:01.301888 12659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33177 every 8 connection(s)
I20260619 16:38:01.302784 12592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:01.305428 12660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:01.310508 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Bootstrap starting.
I20260619 16:38:01.310623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 12592
I20260619 16:38:01.310775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:38:01.311062 392 external_mini_cluster.cc:1428] Setting key 4b5bb52a2bd35d41388129b60195d1a7
I20260619 16:38:01.312934 12660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:01.313709 12660 log.cc:824] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:01.315421 12660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: No bootstrap required, opened a new log
I20260619 16:38:01.318725 12660 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:01.318949 12660 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:01.319029 12660 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba89934c0f6649658f0328cf4bc949ed, State: Initialized, Role: FOLLOWER
I20260619 16:38:01.319443 12660 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
Jun 19 16:38:01 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:01.319589 12660 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:01.319661 12660 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:01.319767 12660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:01.320748 12660 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:01.321096 12660 leader_election.cc:302] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba89934c0f6649658f0328cf4bc949ed; no voters:
I20260619 16:38:01.321496 12660 leader_election.cc:288] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:38:01.321586 12665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:01.321789 12665 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 LEADER]: Becoming Leader. State: Replica: ba89934c0f6649658f0328cf4bc949ed, State: Running, Role: LEADER
I20260619 16:38:01.322150 12665 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:01.322747 12660 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:01.323331 12667 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba89934c0f6649658f0328cf4bc949ed. Latest consensus state: current_term: 1 leader_uuid: "ba89934c0f6649658f0328cf4bc949ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } } }
I20260619 16:38:01.323532 12667 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:01.323801 12666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba89934c0f6649658f0328cf4bc949ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } } }
I20260619 16:38:01.323895 12666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:01.324338 12674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:01.325737 12663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:01.312348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.0.98.62:33177)
0619 16:38:01.312792 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:01.312806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:01.312839 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:38:01.313640 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:01.313658 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:01.314017 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:01.314340 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:01.315113 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:01.316107 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:01.316948 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:01.317461 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:01.317627 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:01.320246 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:01.320252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:01.320263 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:01.320287 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:01.322392 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:01.322927 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:01.322933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:01.322939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:01.323063 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:01.323412 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:01.323415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:01.323417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:01.323971 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:01.324175 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:01.324774 (+ 599us) server_negotiation.cc:299] Negotiation successful
0619 16:38:01.324977 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":145,"threads_started":1}
I20260619 16:38:01.327473 12674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:01.327837 12660 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:38:01.337749 12674 catalog_manager.cc:1347] Generated new cluster ID: 283f0e8ae0174039958e75e559ab1336
I20260619 16:38:01.337821 12674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:01.347595 12674 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:38:01.348826 12674 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:01.358521 12674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Generated new TSK 0
I20260619 16:38:01.359292 12674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:38:02.020859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33177
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:02.161499 12705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:02.161779 12705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:02.161906 12705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:02.165995 12705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:02.166079 12705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:02.166190 12705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:02.171957 12705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:02.173096 12705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:02.174291 12705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:02.184722 12713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:02.184731 12712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:02.185074 12715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:02.186619 12705 server_base.cc:1034] running on GCE node
I20260619 16:38:02.187016 12705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:02.187712 12705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:02.188925 12705 hybrid_clock.cc:648] HybridClock initialized: now 1781887082188854 us; error 91 us; skew 500 ppm
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:02.191654 12705 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:02.192600 12705 webserver.cc:460] Webserver started at http://127.0.98.1:37139/ using document root <none> and password file <none>
I20260619 16:38:02.193168 12705 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:02.193264 12705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:02.193511 12705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:02.209122 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "ac8f7005e46f4b22a1a753f38972716d"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "5498208c97942bdab8db55cd1439f522"
server_key_iv: "bf17b5f636f6fe225fd40b0ab5b6e971"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.209861 12705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ac8f7005e46f4b22a1a753f38972716d"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "5498208c97942bdab8db55cd1439f522"
server_key_iv: "bf17b5f636f6fe225fd40b0ab5b6e971"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.218753 12705 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260619 16:38:02.249735 12722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.251600 12705 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.004s sys 0.000s
I20260619 16:38:02.251737 12705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "ac8f7005e46f4b22a1a753f38972716d"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "5498208c97942bdab8db55cd1439f522"
server_key_iv: "bf17b5f636f6fe225fd40b0ab5b6e971"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.251871 12705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:02.265636 12705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:02.269369 12705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:02.269630 12705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:02.270427 12705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:02.271560 12705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:02.271648 12705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.271788 12705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:02.271875 12705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.283821 12705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39633
I20260619 16:38:02.286263 12705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:38:02.289364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 12705
I20260619 16:38:02.289489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:38:02.289580 12837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39633 every 8 connection(s)
I20260619 16:38:02.289809 392 external_mini_cluster.cc:1428] Setting key 7eb20aa6bdbe01f092f17fe73e13df08
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:02.329588 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:02.292881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57307 (local address 127.0.98.62:33177)
0619 16:38:02.297517 (+ 4636us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:02.297537 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:02.301694 (+ 4157us) server_negotiation.cc:407] Connection header received
0619 16:38:02.301750 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:02.301753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:02.301810 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:02.301909 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:02.313993 (+ 12084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.314850 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.318162 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.318520 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.318604 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:02.324975 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:02.324979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:02.324982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:02.325014 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:02.326993 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:02.327654 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:02.327658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:02.327660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:02.327750 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:02.328187 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:02.328191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:02.328193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:02.328424 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:02.328509 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:02.329228 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 16:38:02.329390 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4538,"thread_start_us":143,"threads_started":1}
I20260619 16:38:02.331421 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:02.291787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33177 (local address 127.0.98.1:57307)
0619 16:38:02.297646 (+ 5859us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:02.297717 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:02.299095 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:02.312603 (+ 13508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:02.312616 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:02.313037 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:02.313811 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:02.313834 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.316547 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.316551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:02.317995 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:02.318008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.318828 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.318832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:02.319043 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:02.319768 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:02.319789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:02.324749 (+ 4960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:02.327159 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:02.327165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:02.327183 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:02.327537 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:02.327858 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:02.327861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:02.327863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:02.328071 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:02.328579 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:02.328585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:02.328937 (+ 352us) client_negotiation.cc:769] Sending connection context
0619 16:38:02.330057 (+ 1120us) client_negotiation.cc:240] Negotiation successful
0619 16:38:02.330398 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5606,"thread_start_us":140,"threads_started":1}
I20260619 16:38:02.333364 12838 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33177
I20260619 16:38:02.333598 12838 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:02.334160 12838 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:02.335855 12624 ts_manager.cc:194] Registered new tserver with Master: ac8f7005e46f4b22a1a753f38972716d (127.0.98.1:39633)
I20260619 16:38:02.337499 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:38:02.356774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33177
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 16:38:02.518216 12844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:02.518599 12844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:02.518678 12844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:02.525161 12844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:02.525233 12844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:02.525341 12844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:02.533377 12844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:02.535009 12844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:02.536402 12844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:02.549937 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:02.549937 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:02.550900 12844 server_base.cc:1034] running on GCE node
W20260619 16:38:02.551246 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:02.551715 12844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:02.552455 12844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:02.553632 12844 hybrid_clock.cc:648] HybridClock initialized: now 1781887082553552 us; error 93 us; skew 500 ppm
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:02.557263 12844 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:02.558686 12844 webserver.cc:460] Webserver started at http://127.0.98.2:40005/ using document root <none> and password file <none>
I20260619 16:38:02.559393 12844 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:02.559523 12844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:02.559782 12844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:02.580309 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "9363e9c6d6aa413685d0bf6f58c886ad"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "e0aec7f2441f5a83ef74ee62120617e0"
server_key_iv: "996589df23a60a54e8ba14fadac5ebee"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.581673 12844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9363e9c6d6aa413685d0bf6f58c886ad"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "e0aec7f2441f5a83ef74ee62120617e0"
server_key_iv: "996589df23a60a54e8ba14fadac5ebee"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.586628 12844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 16:38:02.618160 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.620671 12844 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.004s sys 0.000s
I20260619 16:38:02.620934 12844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "9363e9c6d6aa413685d0bf6f58c886ad"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "e0aec7f2441f5a83ef74ee62120617e0"
server_key_iv: "996589df23a60a54e8ba14fadac5ebee"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.621239 12844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:02.646617 12844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:02.650208 12844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:02.650434 12844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:02.651103 12844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:02.652331 12844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:02.652442 12844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.652585 12844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:02.652667 12844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.662832 12844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34483
I20260619 16:38:02.664533 12844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:38:02.667712 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34483 every 8 connection(s)
I20260619 16:38:02.669693 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 12844
I20260619 16:38:02.669816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:38:02.670169 392 external_mini_cluster.cc:1428] Setting key ca84edd86e3570a9c55ec448382c3dca
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:38:02.695170 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:02.671972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56349 (local address 127.0.98.62:33177)
0619 16:38:02.672103 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:02.672107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:02.676292 (+ 4185us) server_negotiation.cc:407] Connection header received
0619 16:38:02.677647 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:02.677654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:02.677712 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:02.677772 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:02.679586 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.680438 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.682215 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.682592 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.682677 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:02.690532 (+ 7855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:02.690536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:02.690539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:02.690572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:02.692348 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:02.693015 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:02.693019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:02.693021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:02.693117 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:02.693687 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:02.693691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:02.693692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:02.693919 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:02.694006 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:02.694792 (+ 786us) server_negotiation.cc:299] Negotiation successful
0619 16:38:02.694970 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:38:02.696816 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:02.670882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33177 (local address 127.0.98.2:56349)
0619 16:38:02.676086 (+ 5204us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:02.676153 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:02.677374 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:02.678281 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:02.678293 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:02.678682 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:02.679387 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:02.679410 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.680618 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.680621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:02.682068 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:02.682080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.682929 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.682933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:02.683156 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:02.688500 (+ 5344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:02.688523 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:02.690318 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:02.692493 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:02.692499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:02.692515 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:02.692856 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:02.693238 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:02.693244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:02.693247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:02.693564 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:02.694088 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:02.694095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:02.694502 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 16:38:02.695485 (+ 983us) client_negotiation.cc:240] Negotiation successful
0619 16:38:02.695828 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4953,"thread_start_us":139,"threads_started":1}
I20260619 16:38:02.698701 12974 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33177
I20260619 16:38:02.698971 12974 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:02.699587 12974 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:02.700701 12624 ts_manager.cc:194] Registered new tserver with Master: 9363e9c6d6aa413685d0bf6f58c886ad (127.0.98.2:34483)
I20260619 16:38:02.701610 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:38:02.714854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33177
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:02.850885 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:02.851174 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:02.851259 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:02.855636 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:02.855728 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:02.855844 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:02.862576 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:02.864120 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:02.865509 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:02.876655 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:02.876771 12984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:02.876695 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:02.878041 12979 server_base.cc:1034] running on GCE node
I20260619 16:38:02.878435 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:02.879158 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:02.880342 12979 hybrid_clock.cc:648] HybridClock initialized: now 1781887082880316 us; error 43 us; skew 500 ppm
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:02.883181 12979 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:02.884111 12979 webserver.cc:460] Webserver started at http://127.0.98.3:43631/ using document root <none> and password file <none>
I20260619 16:38:02.884594 12979 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:02.884640 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:02.884785 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:02.908846 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "6852bff44b4f4b8ebcdfc273744e5bbf"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "2d29f05a4689bab3532492d5be6a8488"
server_key_iv: "d9be42796b602c79dc0f32c0fe7283d9"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.909564 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6852bff44b4f4b8ebcdfc273744e5bbf"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "2d29f05a4689bab3532492d5be6a8488"
server_key_iv: "d9be42796b602c79dc0f32c0fe7283d9"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.914273 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:02.949002 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.950749 12979 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.001s sys 0.000s
I20260619 16:38:02.950896 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "6852bff44b4f4b8ebcdfc273744e5bbf"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "2d29f05a4689bab3532492d5be6a8488"
server_key_iv: "d9be42796b602c79dc0f32c0fe7283d9"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:02.951063 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:02.965147 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:02.968370 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:02.968544 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:02.969206 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:02.970240 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:02.970300 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.970355 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:02.970382 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:02.980261 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40433
I20260619 16:38:02.981305 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:38:02.987713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 12979
I20260619 16:38:02.987843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:38:02.988166 392 external_mini_cluster.cc:1428] Setting key 0703da706ca39099790eb8ff9440aea2
I20260619 16:38:02.994032 13107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40433 every 8 connection(s)
Jun 19 16:38:02 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:03.007309 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:02.995646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60451 (local address 127.0.98.62:33177)
0619 16:38:02.997008 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:02.997015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:02.997030 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:38:02.997788 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:02.997791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:02.997845 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:02.997920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:02.998947 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.999558 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.000787 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.001123 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.001201 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:03.003770 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:03.003774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:03.003777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:03.003805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:03.005261 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:03.005809 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:03.005814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:03.005816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:03.005896 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:03.006243 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:03.006248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:03.006250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:03.006423 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:03.006547 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:03.007029 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 16:38:03.007149 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1281}
I20260619 16:38:03.008112 13110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:02.996067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33177 (local address 127.0.98.3:60451)
0619 16:38:02.996547 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:02.996598 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:02.997568 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:02.998086 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:02.998092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:02.998350 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:02.998801 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:02.998814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:02.999690 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:02.999694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:03.000649 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:03.000661 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.001420 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.001424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:03.001617 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:03.002259 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:03.002275 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:03.003607 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:03.005418 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:03.005424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:03.005438 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:03.005697 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:03.005991 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:03.005996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:03.005998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:03.006141 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:03.006533 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:03.006540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:03.006789 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 16:38:03.006977 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 16:38:03.007265 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":153,"threads_started":1}
I20260619 16:38:03.009150 13108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33177
I20260619 16:38:03.009399 13108 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.009865 13108 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:03.010852 12624 ts_manager.cc:194] Registered new tserver with Master: 6852bff44b4f4b8ebcdfc273744e5bbf (127.0.98.3:40433)
I20260619 16:38:03.011394 12624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60451
I20260619 16:38:03.013288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:38:03.013890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 12705
I20260619 16:38:03.020035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 12844
I20260619 16:38:03.026535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 12979
I20260619 16:38:03.037611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 12592
I20260619 16:38:03.046353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33177
--webserver_interface=127.0.98.62
--webserver_port=43753
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33177
--ranger_config_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:03.153573 13111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:03.153826 13111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:03.153864 13111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:03.157982 13111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:03.158038 13111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:03.158058 13111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:03.158074 13111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:03.158090 13111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:03.163441 13111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33177
--ranger_config_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:03.164400 13111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:03.165421 13111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:03.173779 13117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:03.173780 13119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:03.173851 13116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:03.173902 13111 server_base.cc:1034] running on GCE node
I20260619 16:38:03.174571 13111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:03.175686 13111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:03.176893 13111 hybrid_clock.cc:648] HybridClock initialized: now 1781887083176865 us; error 43 us; skew 500 ppm
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:03.179924 13111 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:03.180776 13111 webserver.cc:460] Webserver started at http://127.0.98.62:43753/ using document root <none> and password file <none>
I20260619 16:38:03.181269 13111 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:03.181397 13111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:03.232270 13111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:38:03.234659 13127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.236016 13111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:03.236119 13111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "ba89934c0f6649658f0328cf4bc949ed"
format_stamp: "Formatted at 2026-06-19 16:38:01 on dist-test-slave-0f28"
server_key: "61719f0001f9776b12ab039c2bbffb8d"
server_key_iv: "56fc77d25371d0d30d977074ca8ae6e5"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:03.236706 13111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:03.257560 13111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:03.260033 13111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:03.260241 13111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:03.267454 13111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33177
I20260619 16:38:03.267474 13179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33177 every 8 connection(s)
I20260619 16:38:03.268491 13111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:03.272696 13180 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:38:03.273243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13111
I20260619 16:38:03.273720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39633
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33177
--webserver_port=37139
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:03.274411 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Bootstrap starting.
I20260619 16:38:03.290568 13180 log.cc:824] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:03.297209 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:03.297778 13180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Bootstrap complete.
I20260619 16:38:03.301855 13180 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:03.302271 13180 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba89934c0f6649658f0328cf4bc949ed, State: Initialized, Role: FOLLOWER
I20260619 16:38:03.302734 13180 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:03.302875 13180 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:03.302963 13180 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:03.303078 13180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:38:03.304888 13180 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:03.305210 13180 leader_election.cc:302] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba89934c0f6649658f0328cf4bc949ed; no voters:
I20260619 16:38:03.305590 13180 leader_election.cc:288] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:38:03.305644 13184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:38:03.305837 13184 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [term 2 LEADER]: Becoming Leader. State: Replica: ba89934c0f6649658f0328cf4bc949ed, State: Running, Role: LEADER
I20260619 16:38:03.306125 13184 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } }
I20260619 16:38:03.306770 13180 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:03.306972 13185 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba89934c0f6649658f0328cf4bc949ed. Latest consensus state: current_term: 2 leader_uuid: "ba89934c0f6649658f0328cf4bc949ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } } }
I20260619 16:38:03.307008 13186 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba89934c0f6649658f0328cf4bc949ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba89934c0f6649658f0328cf4bc949ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33177 } } }
I20260619 16:38:03.307087 13185 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:03.307114 13186 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:03.307436 13188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:03.310694 13188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:03.311331 13180 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 16:38:03.311326 13188 catalog_manager.cc:1259] Loaded cluster ID: 283f0e8ae0174039958e75e559ab1336
I20260619 16:38:03.312266 13188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:03.318112 13188 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:03.318913 13188 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Loaded TSK: 0
I20260619 16:38:03.320465 13188 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:38:03.410530 13182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:03.410785 13182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:03.410858 13182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:03.417075 13182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:03.417140 13182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:03.417220 13182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:03.423635 13182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37139
--tserver_master_addrs=127.0.98.62:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:03.424647 13182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:03.425678 13182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:03.436234 13218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:03.436239 13220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:03.436779 13182 server_base.cc:1034] running on GCE node
W20260619 16:38:03.436856 13217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:03.437222 13182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:03.437897 13182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:03.439116 13182 hybrid_clock.cc:648] HybridClock initialized: now 1781887083439083 us; error 46 us; skew 500 ppm
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:03.442436 13182 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:03.443624 13182 webserver.cc:460] Webserver started at http://127.0.98.1:37139/ using document root <none> and password file <none>
I20260619 16:38:03.444173 13182 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:03.444281 13182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:03.472370 13182 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260619 16:38:03.475724 13228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.477571 13182 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260619 16:38:03.477705 13182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "ac8f7005e46f4b22a1a753f38972716d"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "5498208c97942bdab8db55cd1439f522"
server_key_iv: "bf17b5f636f6fe225fd40b0ab5b6e971"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:03.478436 13182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:03.517128 13182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:03.520262 13182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:03.520483 13182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:03.521166 13182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:03.522162 13182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:03.522225 13182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.522289 13182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:03.522315 13182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.538548 13182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39633
I20260619 16:38:03.539786 13182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:38:03.541828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13182
I20260619 16:38:03.542315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34483
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33177
--webserver_port=40005
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:03.544658 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39633 every 8 connection(s)
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:03.561868 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:03.542846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33177 (local address 127.0.98.1:45911)
0619 16:38:03.543322 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:03.543540 (+ 218us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:03.544615 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:03.547153 (+ 2538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:03.547163 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:03.547508 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:03.548175 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:03.548194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.550420 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.550424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:03.551791 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:03.551802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.552809 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.552813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:03.553025 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:03.553701 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:03.553721 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:03.556127 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:03.558574 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:03.558580 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:03.558593 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:03.558903 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:03.559289 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:03.559292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:03.559294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:03.559454 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:03.560144 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:03.560151 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:03.560423 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 16:38:03.560612 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 16:38:03.560895 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":156,"threads_started":1}
I20260619 16:38:03.563757 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:03.544170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45911 (local address 127.0.98.62:33177)
0619 16:38:03.545859 (+ 1689us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:03.545883 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:03.545931 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:38:03.546238 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:03.546252 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:03.546693 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:03.548410 (+ 1717us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:03.548702 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.550265 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.551939 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.552638 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.553429 (+ 791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:03.556362 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:03.556368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:03.556383 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:03.556406 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:03.558426 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:03.559047 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:03.559053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:03.559058 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:03.559183 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:03.559593 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:03.559597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:03.559598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:03.560012 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:03.562516 (+ 2504us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:03.562670 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 16:38:03.562935 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":113,"threads_started":1}
I20260619 16:38:03.566146 13342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33177
I20260619 16:38:03.566370 13342 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.566848 13342 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
W20260619 16:38:03.568835 13342 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33177 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:38:03.568962 13342 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.569128 13342 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:03.569967 13342 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.570132 13342 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
W20260619 16:38:03.570871 13342 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:38:03.690727 13345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:03.690984 13345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:03.691054 13345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:03.695113 13345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:03.695194 13345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:03.695287 13345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:03.700477 13345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40005
--tserver_master_addrs=127.0.98.62:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:03.701515 13345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:03.702744 13345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:03.713866 13354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:03.713883 13352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:03.714254 13345 server_base.cc:1034] running on GCE node
W20260619 16:38:03.713881 13351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:03.714735 13345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:03.715396 13345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:03.716588 13345 hybrid_clock.cc:648] HybridClock initialized: now 1781887083716556 us; error 49 us; skew 500 ppm
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:03.719277 13345 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:03.720216 13345 webserver.cc:460] Webserver started at http://127.0.98.2:40005/ using document root <none> and password file <none>
I20260619 16:38:03.720853 13345 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:03.720963 13345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:03.747359 13345 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260619 16:38:03.751644 13362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.753264 13345 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260619 16:38:03.753412 13345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "9363e9c6d6aa413685d0bf6f58c886ad"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "e0aec7f2441f5a83ef74ee62120617e0"
server_key_iv: "996589df23a60a54e8ba14fadac5ebee"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:03.754151 13345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:03.788787 13345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:03.792501 13345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:03.792692 13345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:03.793449 13345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:03.794445 13345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:03.794510 13345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.794564 13345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:03.794595 13345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:03.804950 13345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34483
I20260619 16:38:03.805091 13475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34483 every 8 connection(s)
I20260619 16:38:03.806193 13345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:38:03.812489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13345
I20260619 16:38:03.813055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56763/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40433
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33177
--webserver_port=43631
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:03.828651 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:03.810150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56213 (local address 127.0.98.62:33177)
0619 16:38:03.810287 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:03.810292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:03.810822 (+ 530us) server_negotiation.cc:407] Connection header received
0619 16:38:03.814984 (+ 4162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:03.814988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:03.815057 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:03.815120 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:03.816739 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.817581 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.819384 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.819766 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.819874 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:03.824134 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:03.824138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:03.824143 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:03.824174 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:03.826024 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:03.826714 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:03.826717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:03.826719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:03.826812 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:03.827258 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:03.827263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:03.827265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:03.827539 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:03.827615 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:03.828319 (+ 704us) server_negotiation.cc:299] Negotiation successful
0619 16:38:03.828467 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:38:03.830513 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:03.808862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33177 (local address 127.0.98.2:56213)
0619 16:38:03.810669 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:03.810723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:03.811800 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:03.815541 (+ 3741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:03.815549 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:03.815898 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:03.816563 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:03.816582 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.817790 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.817793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:03.819228 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:03.819239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:03.820847 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:03.820852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:03.821058 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:03.822107 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:03.822127 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:03.823923 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:03.826176 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:03.826182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:03.826200 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:03.826595 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:03.826920 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:03.826924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:03.826926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:03.827131 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:03.827701 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:03.827708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:03.828026 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 16:38:03.829105 (+ 1079us) client_negotiation.cc:240] Negotiation successful
0619 16:38:03.829482 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"thread_start_us":124,"threads_started":1}
I20260619 16:38:03.831754 13476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33177
I20260619 16:38:03.832011 13476 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.832644 13476 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
W20260619 16:38:03.834405 13476 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33177 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:38:03.834525 13476 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.834692 13476 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:03.835489 13476 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:03.835659 13476 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
W20260619 16:38:03.836360 13476 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:38:03.950670 13480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:03.951020 13480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:03.951122 13480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:03.955381 13480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:03.955468 13480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:03.955574 13480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:03.961063 13480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56763/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43631
--tserver_master_addrs=127.0.98.62:33177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:03.962375 13480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:03.963649 13480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:03.974596 13488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:03.974581 13486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:03.974639 13485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:03.974975 13480 server_base.cc:1034] running on GCE node
I20260619 16:38:03.975523 13480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:03.976143 13480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:03.977380 13480 hybrid_clock.cc:648] HybridClock initialized: now 1781887083977341 us; error 53 us; skew 500 ppm
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:03.980266 13480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:03.981106 13480 webserver.cc:460] Webserver started at http://127.0.98.3:43631/ using document root <none> and password file <none>
I20260619 16:38:03.981782 13480 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:03.981916 13480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 16:38:03 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:04.006089 13480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:38:04.009261 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:04.010915 13480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:38:04.011049 13480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "6852bff44b4f4b8ebcdfc273744e5bbf"
format_stamp: "Formatted at 2026-06-19 16:38:02 on dist-test-slave-0f28"
server_key: "2d29f05a4689bab3532492d5be6a8488"
server_key_iv: "d9be42796b602c79dc0f32c0fe7283d9"
server_key_version: "kuduclusterkey@0"
I20260619 16:38:04.011792 13480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:04.036929 13480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:04.040763 13480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:04.041071 13480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:04.041963 13480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:04.042980 13480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:04.043045 13480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:04.043107 13480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:04.043143 13480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:04.055024 13480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40433
I20260619 16:38:04.055160 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40433 every 8 connection(s)
I20260619 16:38:04.056262 13480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:38:04.060886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13480
Jun 19 16:38:04 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:04.071244 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:04.058264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46895 (local address 127.0.98.62:33177)
0619 16:38:04.058446 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:04.058451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:04.059327 (+ 876us) server_negotiation.cc:407] Connection header received
0619 16:38:04.060590 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:04.060593 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:04.060667 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:04.060772 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:04.062133 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:04.062660 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:04.063767 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:04.063986 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:04.064099 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:04.066974 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:04.066977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:04.066979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:04.066999 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:04.068775 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:04.069445 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:04.069449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:04.069451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:04.069546 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:04.070007 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:04.070010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:04.070012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:04.070209 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:04.070309 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:04.070963 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 16:38:04.071083 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 16:38:04.072209 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:04.058616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33177 (local address 127.0.98.3:46895)
0619 16:38:04.059174 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:04.059229 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:04.060288 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:04.060964 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:04.060975 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:04.061330 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:04.061979 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:04.061995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:04.062788 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:04.062791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:04.063638 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:04.063647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:04.064144 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:04.064148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:04.064306 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:04.065014 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:04.065031 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:04.066762 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:04.068918 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:04.068925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:04.068939 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:04.069277 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:04.069656 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:04.069659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:04.069661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:04.069876 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:04.070328 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:04.070335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:04.070665 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 16:38:04.070880 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 16:38:04.071194 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":126,"threads_started":1}
I20260619 16:38:04.075747 13610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33177
I20260619 16:38:04.076067 13610 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:04.076678 13610 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:04.077548 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:04.061606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.98.62:33177)
0619 16:38:04.065551 (+ 3945us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:04.065558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:04.065686 (+ 128us) server_negotiation.cc:407] Connection header received
0619 16:38:04.066601 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:04.066604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:04.066660 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:04.066762 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:04.067895 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:04.069267 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:04.070356 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:04.070576 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:04.070687 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:04.073564 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:04.073568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:04.073571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:04.073594 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:04.075307 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:04.075822 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:04.075827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:04.075829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:04.075915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:04.076219 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:04.076223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:04.076224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:04.076416 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:04.076612 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:04.077287 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 16:38:04.077421 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3860,"thread_start_us":86,"threads_started":1}
W20260619 16:38:04.078580 13610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33177 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 16:38:04.078727 13610 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:04.078923 13610 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:04.079674 13610 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:04.079864 13610 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
W20260619 16:38:04.080595 13610 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 16:38:04 dist-test-slave-0f28 krb5kdc[12148](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781887084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:04.571111 13342 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:04.571419 13342 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:04.572561 13144 ts_manager.cc:194] Registered new tserver with Master: ac8f7005e46f4b22a1a753f38972716d (127.0.98.1:39633)
I20260619 16:38:04.574258 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45911
Jun 19 16:38:04 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:04.836583 13476 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:04.836876 13476 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:04.837632 13144 ts_manager.cc:194] Registered new tserver with Master: 9363e9c6d6aa413685d0bf6f58c886ad (127.0.98.2:34483)
I20260619 16:38:04.838459 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56213
Jun 19 16:38:05 dist-test-slave-0f28 krb5kdc[12148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 16:38:05.080832 13610 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:05.081140 13610 heartbeater.cc:511] Master 127.0.98.62:33177 requested a full tablet report, sending...
I20260619 16:38:05.081851 13144 ts_manager.cc:194] Registered new tserver with Master: 6852bff44b4f4b8ebcdfc273744e5bbf (127.0.98.3:40433)
I20260619 16:38:05.082629 13144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46895
I20260619 16:38:05.090693 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:38:05.112694 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.098136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.0.98.62:33177)
0619 16:38:05.100024 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.100029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.100262 (+ 233us) server_negotiation.cc:407] Connection header received
0619 16:38:05.100451 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.100454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.100503 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.100567 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:05.101563 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.102427 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.103645 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.103998 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.104092 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.105992 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:05.105995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:05.105998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:05.106037 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:05.107917 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:05.108447 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:05.108452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:05.108454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:05.108561 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:05.108912 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:05.108916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:05.108918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:05.109166 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:05.109311 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.112201 (+ 2890us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.112344 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1789,"thread_start_us":115,"threads_started":1}
I20260619 16:38:05.118193 13144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:38:05.123126 13144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:38:05.171191 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.158364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40433 (local address 127.0.0.1:46010)
0619 16:38:05.164205 (+ 5841us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:05.164218 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:05.164298 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:05.166369 (+ 2071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:05.166372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:05.166389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:05.166486 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.166505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.167886 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.167889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.169307 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.169336 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.170649 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.170653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.170768 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.170807 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 16:38:05.170916 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:38:05.171030 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5778,"thread_start_us":94,"threads_started":1}
I20260619 16:38:05.171856 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.156448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34483 (local address 127.0.0.1:37664)
0619 16:38:05.163768 (+ 7320us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:05.163824 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:05.163960 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:05.165229 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:05.165235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:05.165258 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:05.165413 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.165420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.168998 (+ 3578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.169002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.170184 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.170195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.171465 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.171468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.171560 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.171584 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:38:05.171650 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:38:05.171717 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7131,"thread_start_us":123,"threads_started":1}
I20260619 16:38:05.173887 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.158483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.0.98.3:40433)
0619 16:38:05.165826 (+ 7343us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.165833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.165855 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:38:05.165923 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.165927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.166099 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.166235 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:05.166638 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.167762 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.169463 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.170310 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.170416 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.170569 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.173641 (+ 3072us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.173748 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7231,"thread_start_us":121,"threads_started":1}
I20260619 16:38:05.177932 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.157282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.0.98.2:34483)
0619 16:38:05.164527 (+ 7245us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.164534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.164559 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:38:05.164638 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.164643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.164861 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.165009 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:05.167577 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.168852 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.170334 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.171159 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.171270 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.171402 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.177655 (+ 6253us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.177772 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7143,"thread_start_us":103,"threads_started":1}
I20260619 16:38:05.180951 13410 tablet_service.cc:1467] Processing CreateTablet for tablet 40d6ad71065d47aa94bc2c0020ff1cd8 (DEFAULT_TABLE table=test-table [id=a2444fa7258640af8196bbcf3e28800c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:05.180953 13544 tablet_service.cc:1467] Processing CreateTablet for tablet 40d6ad71065d47aa94bc2c0020ff1cd8 (DEFAULT_TABLE table=test-table [id=a2444fa7258640af8196bbcf3e28800c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:05.181473 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.158746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39633 (local address 127.0.0.1:36912)
0619 16:38:05.165458 (+ 6712us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:05.165473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:05.165572 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:05.175819 (+ 10247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:05.175822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:05.175845 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:05.175955 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.175963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.178917 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.178921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.180002 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.180023 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.181085 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.181089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.181172 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.181197 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:38:05.181244 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 16:38:05.181306 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6649,"thread_start_us":83,"threads_started":1}
I20260619 16:38:05.182070 13410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d6ad71065d47aa94bc2c0020ff1cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:05.182071 13544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d6ad71065d47aa94bc2c0020ff1cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:05.182178 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.158935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.0.98.1:39633)
0619 16:38:05.172265 (+ 13330us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.172271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.172296 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:38:05.172370 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.172375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.172572 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.176153 (+ 3581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:05.176324 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.177500 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.180171 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.180926 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.181728 (+ 802us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.181883 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.181930 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.182031 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13220,"thread_start_us":61,"threads_started":1}
I20260619 16:38:05.184628 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 40d6ad71065d47aa94bc2c0020ff1cd8 (DEFAULT_TABLE table=test-table [id=a2444fa7258640af8196bbcf3e28800c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:05.185551 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d6ad71065d47aa94bc2c0020ff1cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:05.189638 13645 tablet_bootstrap.cc:492] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Bootstrap starting.
I20260619 16:38:05.191642 13645 tablet_bootstrap.cc:654] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:05.192535 13645 log.cc:824] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:05.195909 13648 tablet_bootstrap.cc:492] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: Bootstrap starting.
I20260619 16:38:05.198315 13648 tablet_bootstrap.cc:654] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:05.199159 13648 log.cc:824] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:05.200979 13645 tablet_bootstrap.cc:492] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: No bootstrap required, opened a new log
I20260619 16:38:05.201203 13645 ts_tablet_manager.cc:1397] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20260619 16:38:05.204726 13646 tablet_bootstrap.cc:492] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Bootstrap starting.
I20260619 16:38:05.205593 13645 raft_consensus.cc:348] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.205879 13645 raft_consensus.cc:374] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:05.205936 13645 raft_consensus.cc:729] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6852bff44b4f4b8ebcdfc273744e5bbf, State: Initialized, Role: FOLLOWER
I20260619 16:38:05.206564 13645 consensus_queue.cc:260] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.207764 13645 ts_tablet_manager.cc:1428] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 16:38:05.208145 13610 heartbeater.cc:503] Master 127.0.98.62:33177 was elected leader, sending a full tablet report...
I20260619 16:38:05.208482 13646 tablet_bootstrap.cc:654] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:05.209365 13646 log.cc:824] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:05.213570 13648 tablet_bootstrap.cc:492] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: No bootstrap required, opened a new log
I20260619 16:38:05.213788 13648 ts_tablet_manager.cc:1397] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260619 16:38:05.218561 13648 raft_consensus.cc:348] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.218845 13648 raft_consensus.cc:374] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:05.218911 13648 raft_consensus.cc:729] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8f7005e46f4b22a1a753f38972716d, State: Initialized, Role: FOLLOWER
I20260619 16:38:05.219426 13648 consensus_queue.cc:260] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.220477 13648 ts_tablet_manager.cc:1428] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 16:38:05.220826 13342 heartbeater.cc:503] Master 127.0.98.62:33177 was elected leader, sending a full tablet report...
I20260619 16:38:05.223198 13646 tablet_bootstrap.cc:492] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: No bootstrap required, opened a new log
I20260619 16:38:05.223400 13646 ts_tablet_manager.cc:1397] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260619 16:38:05.227689 13646 raft_consensus.cc:348] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.227967 13646 raft_consensus.cc:374] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:05.228037 13646 raft_consensus.cc:729] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9363e9c6d6aa413685d0bf6f58c886ad, State: Initialized, Role: FOLLOWER
I20260619 16:38:05.228572 13646 consensus_queue.cc:260] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.229588 13646 ts_tablet_manager.cc:1428] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:38:05.229954 13476 heartbeater.cc:503] Master 127.0.98.62:33177 was elected leader, sending a full tablet report...
W20260619 16:38:05.292366 13343 tablet.cc:2369] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:38:05.308079 13611 tablet.cc:2369] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:38:05.316051 13477 tablet.cc:2369] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:05.340005 13650 raft_consensus.cc:482] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:05.340266 13650 raft_consensus.cc:504] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.341686 13650 leader_election.cc:288] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9363e9c6d6aa413685d0bf6f58c886ad (127.0.98.2:34483), ac8f7005e46f4b22a1a753f38972716d (127.0.98.1:39633)
I20260619 16:38:05.345964 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.342057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54113 (local address 127.0.98.2:34483)
0619 16:38:05.342239 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.342245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.342327 (+ 82us) server_negotiation.cc:407] Connection header received
0619 16:38:05.342537 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.342540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.342591 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.342658 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:05.343131 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.343730 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.344902 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.345306 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.345459 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.345521 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.345765 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.345841 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 16:38:05.345970 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.341933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34483 (local address 127.0.98.3:54113)
0619 16:38:05.342238 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:05.342254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:05.342405 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:05.342813 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:05.342819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:05.342837 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:05.342995 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.343001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.343842 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.343847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.344770 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.344779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.345472 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.345476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.345593 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.345645 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 16:38:05.345727 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 16:38:05.345798 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260619 16:38:05.346935 13430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d6ad71065d47aa94bc2c0020ff1cd8" candidate_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" is_pre_election: true
I20260619 16:38:05.347026 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.342321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39633 (local address 127.0.98.3:53615)
0619 16:38:05.342767 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:05.342781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:05.342866 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:05.343142 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:05.343145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:05.343162 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:05.343266 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.343272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.344287 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.344290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.345543 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:05.345554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.346651 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.346654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:05.346724 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.346741 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:38:05.346838 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 16:38:05.346898 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":92,"threads_started":1}
I20260619 16:38:05.347025 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.342478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53615 (local address 127.0.98.1:39633)
0619 16:38:05.342595 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.342599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.342886 (+ 287us) server_negotiation.cc:407] Connection header received
0619 16:38:05.342955 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.342958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.343013 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.343091 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:05.343394 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.344152 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.345679 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.346278 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.346387 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.346446 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.346855 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.346921 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:38:05.347231 13430 raft_consensus.cc:2454] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6852bff44b4f4b8ebcdfc273744e5bbf in term 0.
I20260619 16:38:05.347721 13498 leader_election.cc:302] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6852bff44b4f4b8ebcdfc273744e5bbf, 9363e9c6d6aa413685d0bf6f58c886ad; no voters:
I20260619 16:38:05.347978 13650 raft_consensus.cc:2781] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:38:05.348026 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d6ad71065d47aa94bc2c0020ff1cd8" candidate_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8f7005e46f4b22a1a753f38972716d" is_pre_election: true
I20260619 16:38:05.348083 13650 raft_consensus.cc:482] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:38:05.348138 13650 raft_consensus.cc:3037] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:05.348372 13296 raft_consensus.cc:2454] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6852bff44b4f4b8ebcdfc273744e5bbf in term 0.
I20260619 16:38:05.349409 13650 raft_consensus.cc:504] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.349769 13650 leader_election.cc:288] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [CANDIDATE]: Term 1 election: Requested vote from peers 9363e9c6d6aa413685d0bf6f58c886ad (127.0.98.2:34483), ac8f7005e46f4b22a1a753f38972716d (127.0.98.1:39633)
I20260619 16:38:05.350085 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d6ad71065d47aa94bc2c0020ff1cd8" candidate_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8f7005e46f4b22a1a753f38972716d"
I20260619 16:38:05.350064 13430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d6ad71065d47aa94bc2c0020ff1cd8" candidate_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9363e9c6d6aa413685d0bf6f58c886ad"
I20260619 16:38:05.350200 13296 raft_consensus.cc:3037] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:05.350201 13430 raft_consensus.cc:3037] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:05.351405 13430 raft_consensus.cc:2454] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6852bff44b4f4b8ebcdfc273744e5bbf in term 1.
I20260619 16:38:05.351405 13296 raft_consensus.cc:2454] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6852bff44b4f4b8ebcdfc273744e5bbf in term 1.
I20260619 16:38:05.351709 13498 leader_election.cc:302] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6852bff44b4f4b8ebcdfc273744e5bbf, ac8f7005e46f4b22a1a753f38972716d; no voters:
I20260619 16:38:05.351872 13650 raft_consensus.cc:2781] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:05.352064 13650 raft_consensus.cc:686] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 1 LEADER]: Becoming Leader. State: Replica: 6852bff44b4f4b8ebcdfc273744e5bbf, State: Running, Role: LEADER
I20260619 16:38:05.352393 13650 consensus_queue.cc:237] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } }
I20260619 16:38:05.355863 13144 catalog_manager.cc:5495] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6852bff44b4f4b8ebcdfc273744e5bbf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6852bff44b4f4b8ebcdfc273744e5bbf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40433 } health_report { overall_health: HEALTHY } } }
I20260619 16:38:05.396323 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.392097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.0.98.3:40433)
0619 16:38:05.392251 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:05.392256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:05.392272 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:38:05.392396 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:05.392399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:05.392468 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:05.392589 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:05.392823 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.393644 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.394603 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:05.394948 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:05.395055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:05.395333 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:05.395909 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:38:05.396029 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:05.396105 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:38:05.396163 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:38:05.402401 13296 raft_consensus.cc:1261] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 1 FOLLOWER]: Refusing update from remote peer 6852bff44b4f4b8ebcdfc273744e5bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:05.402474 13430 raft_consensus.cc:1261] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 1 FOLLOWER]: Refusing update from remote peer 6852bff44b4f4b8ebcdfc273744e5bbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:05.403137 13663 consensus_queue.cc:1035] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8f7005e46f4b22a1a753f38972716d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:05.403318 13650 consensus_queue.cc:1035] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9363e9c6d6aa413685d0bf6f58c886ad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:05.417786 13668 mvcc.cc:205] Tried to move back new op lower bound from 7298609501800824832 to 7298609501606195200. Current Snapshot: MvccSnapshot[applied={T|T < 7298609501800824832}]
I20260619 16:38:05.421841 13143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:05.422003 13143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:05.424551 13143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba89934c0f6649658f0328cf4bc949ed: Sending DeleteTablet for 3 replicas of tablet 40d6ad71065d47aa94bc2c0020ff1cd8
I20260619 16:38:05.425612 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d6ad71065d47aa94bc2c0020ff1cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260619 16:38:05.426086 13675 tablet_replica.cc:330] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d: stopping tablet replica
I20260619 16:38:05.426080 13544 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d6ad71065d47aa94bc2c0020ff1cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46010
I20260619 16:38:05.426455 13675 raft_consensus.cc:2229] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:05.426620 13410 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d6ad71065d47aa94bc2c0020ff1cd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37664
I20260619 16:38:05.426741 13675 raft_consensus.cc:2258] T 40d6ad71065d47aa94bc2c0020ff1cd8 P ac8f7005e46f4b22a1a753f38972716d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:05.427589 13677 tablet_replica.cc:330] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: stopping tablet replica
I20260619 16:38:05.427665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13182
I20260619 16:38:05.428337 13677 raft_consensus.cc:2229] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:05.428668 13677 raft_consensus.cc:2258] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:05.430106 13677 ts_tablet_manager.cc:1905] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:05.432719 13677 ts_tablet_manager.cc:1918] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:05.432862 13677 log.cc:1196] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/40d6ad71065d47aa94bc2c0020ff1cd8
I20260619 16:38:05.433292 13677 ts_tablet_manager.cc:1939] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 9363e9c6d6aa413685d0bf6f58c886ad: Deleting consensus metadata
I20260619 16:38:05.434609 13129 catalog_manager.cc:4841] TS 9363e9c6d6aa413685d0bf6f58c886ad (127.0.98.2:34483): tablet 40d6ad71065d47aa94bc2c0020ff1cd8 (table test-table [id=a2444fa7258640af8196bbcf3e28800c]) successfully deleted
W20260619 16:38:05.435039 13129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39633 (error 108)
I20260619 16:38:05.435386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13345
I20260619 16:38:05.437077 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:05.436782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39633 (local address 127.0.0.1:36928)
0619 16:38:05.436936 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:05.436997 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 16:38:05.437239 13129 catalog_manager.cc:4584] TS ac8f7005e46f4b22a1a753f38972716d (127.0.98.1:39633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40d6ad71065d47aa94bc2c0020ff1cd8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39633: connect: Connection refused (error 111)
I20260619 16:38:05.437593 13676 tablet_replica.cc:330] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: stopping tablet replica
I20260619 16:38:05.437858 13676 raft_consensus.cc:2229] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:38:05.438158 13676 raft_consensus.cc:2258] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:05.439962 13676 ts_tablet_manager.cc:1905] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:05.442636 13676 ts_tablet_manager.cc:1918] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:05.442727 13676 log.cc:1196] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/40d6ad71065d47aa94bc2c0020ff1cd8
I20260619 16:38:05.443022 13676 ts_tablet_manager.cc:1939] T 40d6ad71065d47aa94bc2c0020ff1cd8 P 6852bff44b4f4b8ebcdfc273744e5bbf: Deleting consensus metadata
I20260619 16:38:05.444043 13128 catalog_manager.cc:4841] TS 6852bff44b4f4b8ebcdfc273744e5bbf (127.0.98.3:40433): tablet 40d6ad71065d47aa94bc2c0020ff1cd8 (table test-table [id=a2444fa7258640af8196bbcf3e28800c]) successfully deleted
I20260619 16:38:05.444370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13480
I20260619 16:38:05.449519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13111
I20260619 16:38:05.457578 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 16:38:05.802080 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 16:38:05.802210 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 16:38:06.147145 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 16:38:06.147 UTC [12178] LOG: received smart shutdown request
2026-06-19 16:38:06.151 UTC [12178] LOG: background worker "logical replication launcher" (PID 12187) exited with exit code 1
2026-06-19 16:38:06.151 UTC [12182] LOG: shutting down
2026-06-19 16:38:06.340 UTC [12178] LOG: database system is shut down
2026-06-19T16:38:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47342 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:38:06 dist-test-slave-0f28 krb5kdc[13687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:38:06 dist-test-slave-0f28 krb5kdc[13687](info): set up 2 sockets
Jun 19 16:38:06 dist-test-slave-0f28 krb5kdc[13687](info): commencing operation
krb5kdc: starting...
W20260619 16:38:07.517426 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:38:07 dist-test-slave-0f28 krb5kdc[13687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:38:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:38:07.636955 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32993,127.0.98.61:37645,127.0.98.60:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:07.741752 13700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:07.742023 13700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:07.742089 13700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:07.746620 13700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:07.746696 13700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:07.746734 13700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:07.746770 13700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:07.746805 13700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:07.751721 13700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32993,127.0.98.61:37645,127.0.98.60:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:07.752655 13700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:07.753619 13700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:07.762758 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:07.762777 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:07.762809 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:07.763125 13700 server_base.cc:1034] running on GCE node
I20260619 16:38:07.763643 13700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:07.764662 13700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:07.765823 13700 hybrid_clock.cc:648] HybridClock initialized: now 1781887087765792 us; error 44 us; skew 500 ppm
Jun 19 16:38:07 dist-test-slave-0f28 krb5kdc[13687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:07.768628 13700 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:07.769536 13700 webserver.cc:460] Webserver started at http://127.0.98.62:39399/ using document root <none> and password file <none>
I20260619 16:38:07.770052 13700 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:07.770115 13700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:07.770339 13700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:07.772209 13700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "fcaccaeeef884b0a8ff0c1387c916270"
format_stamp: "Formatted at 2026-06-19 16:38:07 on dist-test-slave-0f28"
server_key: "459f2f83d2054e9723ec22f50152f4e9"
server_key_iv: "5edaff3b6d337fb216747214fc8a4944"
server_key_version: "encryptionkey@0"
I20260619 16:38:07.772746 13700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "fcaccaeeef884b0a8ff0c1387c916270"
format_stamp: "Formatted at 2026-06-19 16:38:07 on dist-test-slave-0f28"
server_key: "459f2f83d2054e9723ec22f50152f4e9"
server_key_iv: "5edaff3b6d337fb216747214fc8a4944"
server_key_version: "encryptionkey@0"
I20260619 16:38:07.776163 13700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 16:38:07.778466 13715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:07.779618 13700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:07.779769 13700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "fcaccaeeef884b0a8ff0c1387c916270"
format_stamp: "Formatted at 2026-06-19 16:38:07 on dist-test-slave-0f28"
server_key: "459f2f83d2054e9723ec22f50152f4e9"
server_key_iv: "5edaff3b6d337fb216747214fc8a4944"
server_key_version: "encryptionkey@0"
I20260619 16:38:07.779877 13700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:07.804265 13700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:07.807371 13700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:07.807567 13700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:07.814282 13700 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32993
I20260619 16:38:07.814308 13767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32993 every 8 connection(s)
I20260619 16:38:07.815191 13700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:07.817703 13768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:07.821121 13768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:07.822710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13700
I20260619 16:38:07.822849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:38:07.823120 392 external_mini_cluster.cc:1428] Setting key 6fb505a9f82f64bd09c608df2b78dec3
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Jun 19 16:38:07 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 16:38:07.836526 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.823297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.0.1:36624)
0619 16:38:07.823817 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.823870 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:07.824962 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:07.825814 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:07.825819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:07.826140 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:07.826378 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:07.826390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:07.828155 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:07.828158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:07.828884 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:07.828893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:07.829803 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:07.829806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:07.829896 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:07.830460 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:07.830475 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:07.831851 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:07.833975 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:07.833983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:07.833996 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:07.834211 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:07.834527 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:07.834530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:07.834532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:07.834640 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:07.835293 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:07.835306 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:07.835425 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 16:38:07.835568 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 16:38:07.835799 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260619 16:38:07.836530 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.823297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.98.62:32993)
0619 16:38:07.823789 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:07.823810 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:07.824012 (+ 202us) server_negotiation.cc:407] Connection header received
0619 16:38:07.825272 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:07.825281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:07.825567 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:07.825777 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:07.826534 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:07.828013 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:07.829016 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:07.829651 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:07.829824 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:07.831986 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:07.831991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:07.832001 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:07.832021 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:07.833858 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:07.834321 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:07.834327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:07.834331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:07.834429 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:07.834736 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:07.834740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:07.834741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:07.835136 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:07.835275 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:07.835670 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 16:38:07.835920 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":380,"spinlock_wait_cycles":49536,"thread_start_us":138,"threads_started":1}
I20260619 16:38:07.839143 13768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:07.839866 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.839571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46360)
0619 16:38:07.839707 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.839790 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 16:38:07.840080 13718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
I20260619 16:38:07.840951 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.840694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46376)
0619 16:38:07.840814 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.840873 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 16:38:07.841130 13768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:38:07.850517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37645
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32993,127.0.98.61:37645,127.0.98.60:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:07.890393 13768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } attempt: 1
I20260619 16:38:07.891426 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.891008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46388)
0619 16:38:07.891209 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.891296 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:38:07.892318 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.892069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46402)
0619 16:38:07.892201 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.892254 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:38:07.892482 13768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
W20260619 16:38:07.957748 13775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:07.958030 13775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:07.958115 13775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:38:07.960670 13768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } attempt: 2
I20260619 16:38:07.961560 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.961209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46408)
0619 16:38:07.961399 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.961468 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 16:38:07.962316 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:07.962111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46416)
0619 16:38:07.962214 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:07.962261 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:38:07.962437 13775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:07.962481 13768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37645: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37645: connect: Connection refused (error 111)
W20260619 16:38:07.962534 13775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:07.962589 13775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:07.962641 13775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:07.962673 13775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:07.967804 13775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32993,127.0.98.61:37645,127.0.98.60:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:07.968757 13775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:07.969775 13775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:07.977876 13781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:07.977912 13775 server_base.cc:1034] running on GCE node
W20260619 16:38:07.977869 13783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:07.977869 13780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:07.978613 13775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:07.979681 13775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:07.980885 13775 hybrid_clock.cc:648] HybridClock initialized: now 1781887087980867 us; error 28 us; skew 500 ppm
Jun 19 16:38:07 dist-test-slave-0f28 krb5kdc[13687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:07.983852 13775 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:38:07.984728 13775 webserver.cc:460] Webserver started at http://127.0.98.61:39969/ using document root <none> and password file <none>
I20260619 16:38:07.985258 13775 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:07.985370 13775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:07.985597 13775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:07.987547 13775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data/instance:
uuid: "53beed7eece34ebb8391276b97252e56"
format_stamp: "Formatted at 2026-06-19 16:38:07 on dist-test-slave-0f28"
server_key: "a06d3a8ce62df3cef06de70c0ca8b5d0"
server_key_iv: "f6b63c2866e370040592e4d39e448eec"
server_key_version: "encryptionkey@0"
I20260619 16:38:07.988088 13775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal/instance:
uuid: "53beed7eece34ebb8391276b97252e56"
format_stamp: "Formatted at 2026-06-19 16:38:07 on dist-test-slave-0f28"
server_key: "a06d3a8ce62df3cef06de70c0ca8b5d0"
server_key_iv: "f6b63c2866e370040592e4d39e448eec"
server_key_version: "encryptionkey@0"
I20260619 16:38:07.991564 13775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:38:07.993897 13790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:07.995086 13775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:07.995231 13775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal
uuid: "53beed7eece34ebb8391276b97252e56"
format_stamp: "Formatted at 2026-06-19 16:38:07 on dist-test-slave-0f28"
server_key: "a06d3a8ce62df3cef06de70c0ca8b5d0"
server_key_iv: "f6b63c2866e370040592e4d39e448eec"
server_key_version: "encryptionkey@0"
I20260619 16:38:07.995338 13775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:08.015791 13775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:08.018862 13775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:08.019058 13775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:08.026183 13842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37645 every 8 connection(s)
I20260619 16:38:08.026189 13775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37645
I20260619 16:38:08.027093 13775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:38:08.029721 13843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:08.033128 13843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:08.036430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13775
I20260619 16:38:08.036576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-1/wal/instance
I20260619 16:38:08.036856 392 external_mini_cluster.cc:1428] Setting key 8a4710a6cc07d9e4da47cd2626829ffa
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 16:38:08.046845 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.034874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.0.98.62:32993)
0619 16:38:08.035014 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.035017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.035726 (+ 709us) server_negotiation.cc:407] Connection header received
0619 16:38:08.036773 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.036777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.036842 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.036948 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.038098 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.038630 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.039736 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.039982 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.040093 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.042569 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.042572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.042574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.042595 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.044121 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.044662 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.044668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.044670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.044747 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.045101 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.045105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.045106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.045376 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.045490 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.046573 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.046695 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:38:08.047268 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.035149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.0.1:36636)
0619 16:38:08.035583 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.035622 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.036461 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.037113 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.037120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.037500 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.037953 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.037966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.038771 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.038774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.039611 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.039620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.040142 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.040146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.040359 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.041007 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.041023 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.042405 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.044278 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.044284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.044297 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.044557 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.044839 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.044844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.044846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.044996 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.045501 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.045509 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.045738 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.045919 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.046258 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260619 16:38:08.048620 13843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:08.053699 13768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } attempt: 3
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:38:08.059332 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.049141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46422)
0619 16:38:08.049284 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.049297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.049423 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.049995 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.049998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.050168 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.050237 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.050242 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.051268 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.051274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.051826 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.051834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.052404 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.052407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.052595 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.053117 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.053128 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.054688 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.057006 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.057009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.057011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.057266 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.058111 (+ 845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.058114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.058115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.058225 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.058803 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.058806 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.058895 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.058991 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.059132 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":67}
I20260619 16:38:08.059900 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.049294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.0.98.61:37645)
0619 16:38:08.049647 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.049656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.049674 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:38:08.049705 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.049709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.049839 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.049981 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.050344 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.051157 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.051942 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.052293 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.052408 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.054856 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.054874 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.054884 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.054904 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.056831 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.057432 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.057440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.057445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.057539 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.058318 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.058322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.058324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.058629 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.058760 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.059100 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.059255 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260619 16:38:08.061659 13843 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:38:08.062348 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.062104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59186)
0619 16:38:08.062213 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.062287 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:38:08.062539 13791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
I20260619 16:38:08.062772 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.054124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46424)
0619 16:38:08.054301 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.054317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.054392 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.054724 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.054727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.054885 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.054991 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.054996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.055769 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.055773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.056363 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.056377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.057032 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.057036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.057153 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.057658 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.057679 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.059085 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.061086 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.061089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.061091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.061384 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.061665 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.061669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.061671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.061764 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.062267 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.062270 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.062369 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.062483 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.062623 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 16:38:08.062863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38333
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32993,127.0.98.61:37645,127.0.98.60:38333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:08.063302 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.054161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46424 (local address 127.0.98.61:37645)
0619 16:38:08.054428 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.054438 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.054452 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:38:08.054512 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.054515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.054579 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.054707 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.055102 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.055651 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.056492 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.056885 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.056977 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.059286 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.059289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.059292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.059313 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.060969 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.061479 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.061482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.061484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.061572 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.061875 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.061880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.061884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.062131 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.062270 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.063077 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.063163 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260619 16:38:08.063673 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.063410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59196)
0619 16:38:08.063536 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.063596 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:38:08.063838 13843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38333: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
I20260619 16:38:08.063844 13768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:08.064600 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.064306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59202)
0619 16:38:08.064479 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.064529 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 16:38:08.065289 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.065062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59218)
0619 16:38:08.065201 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.065242 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260619 16:38:08.065490 13768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38333: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
I20260619 16:38:08.096664 13768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } attempt: 1
I20260619 16:38:08.097523 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.097205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59232)
0619 16:38:08.097381 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.097449 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:38:08.098228 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.098013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59246)
0619 16:38:08.098114 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.098160 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:38:08.098352 13768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38333: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
I20260619 16:38:08.113091 13843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } attempt: 1
I20260619 16:38:08.113893 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.113622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59260)
0619 16:38:08.113757 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.113822 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:38:08.114619 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.114407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59264)
0619 16:38:08.114511 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.114558 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:38:08.114768 13843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38333: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
W20260619 16:38:08.171380 13851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:08.171622 13851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:08.171660 13851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 16:38:08.173503 13768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } attempt: 2
I20260619 16:38:08.174414 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.174073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59268)
0619 16:38:08.174253 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.174322 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 16:38:08.175185 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.174966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59276)
0619 16:38:08.175077 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.175129 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:38:08.175328 13768 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38333: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
W20260619 16:38:08.176225 13851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:08.176270 13851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:08.176290 13851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:08.176326 13851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:08.176366 13851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:08.181574 13851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32993,127.0.98.61:37645,127.0.98.60:38333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:08.182524 13851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:08.182921 13843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } attempt: 2
I20260619 16:38:08.183600 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.183330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59290)
0619 16:38:08.183461 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.183537 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 16:38:08.183717 13851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:38:08.184255 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.184052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59306)
0619 16:38:08.184155 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.184199 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:38:08.184398 13843 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38333: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38333: connect: Connection refused (error 111)
W20260619 16:38:08.192008 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:08.192070 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:08.192072 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:08.192256 13851 server_base.cc:1034] running on GCE node
I20260619 16:38:08.192859 13851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:08.194001 13851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:08.195186 13851 hybrid_clock.cc:648] HybridClock initialized: now 1781887088195173 us; error 24 us; skew 500 ppm
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:08.197834 13851 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:38:08.198764 13851 webserver.cc:460] Webserver started at http://127.0.98.60:41851/ using document root <none> and password file <none>
I20260619 16:38:08.199353 13851 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:08.199442 13851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:08.199661 13851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:08.201642 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data/instance:
uuid: "94cdb0d302f942b7b9ae8786db0d1be2"
format_stamp: "Formatted at 2026-06-19 16:38:08 on dist-test-slave-0f28"
server_key: "79f2b0b6ee5c66c111d554904b20079d"
server_key_iv: "48301de583f163ee29f83fa8f4e41124"
server_key_version: "encryptionkey@0"
I20260619 16:38:08.202184 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal/instance:
uuid: "94cdb0d302f942b7b9ae8786db0d1be2"
format_stamp: "Formatted at 2026-06-19 16:38:08 on dist-test-slave-0f28"
server_key: "79f2b0b6ee5c66c111d554904b20079d"
server_key_iv: "48301de583f163ee29f83fa8f4e41124"
server_key_version: "encryptionkey@0"
I20260619 16:38:08.205614 13851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:38:08.207847 13866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:08.209004 13851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:08.209142 13851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal
uuid: "94cdb0d302f942b7b9ae8786db0d1be2"
format_stamp: "Formatted at 2026-06-19 16:38:08 on dist-test-slave-0f28"
server_key: "79f2b0b6ee5c66c111d554904b20079d"
server_key_iv: "48301de583f163ee29f83fa8f4e41124"
server_key_version: "encryptionkey@0"
I20260619 16:38:08.209247 13851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:08.228147 13851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:08.231011 13851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:08.231205 13851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:08.238193 13851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38333
I20260619 16:38:08.238204 13918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38333 every 8 connection(s)
I20260619 16:38:08.239059 13851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:38:08.241771 13919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:08.245733 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:08.249239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 13851
I20260619 16:38:08.249399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781886927079793-392-0/minicluster-data/master-2/wal/instance
I20260619 16:38:08.249742 392 external_mini_cluster.cc:1428] Setting key 53d89a9cc4764ceb3bff7eba610a2db7
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:38:08.259891 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.247587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.0.98.62:32993)
0619 16:38:08.247739 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.247743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.248425 (+ 682us) server_negotiation.cc:407] Connection header received
0619 16:38:08.249479 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.249483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.249545 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.249622 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.251026 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.251650 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.252771 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.253030 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.253108 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.255993 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.255997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.256000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.256027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.257723 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.258358 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.258361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.258362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.258436 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.258788 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.258791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.258792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.259018 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.259122 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.259650 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.259774 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 16:38:08.260818 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.247878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32993 (local address 127.0.0.1:36642)
0619 16:38:08.248287 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.248326 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.249230 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.249812 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.249822 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.250248 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.250887 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.250905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.251775 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.251779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.252639 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.252651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.253172 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.253176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.253387 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.254010 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.254026 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.255845 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.257829 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.257841 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.257851 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.258257 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.258531 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.258534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.258536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.258696 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.259119 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.259124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.259389 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.259581 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.259873 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260619 16:38:08.260836 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.250425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59320 (local address 127.0.98.60:38333)
0619 16:38:08.250731 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.250739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.250759 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:38:08.250884 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.250888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.251064 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.251250 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.251951 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.252745 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.253796 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.254141 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.254255 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.257033 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.257041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.257055 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.257083 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.258869 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.259324 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.259328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.259330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.259405 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.259716 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.259719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.259721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.259964 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.260135 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.260469 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.260574 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260619 16:38:08.262157 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:08.270848 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.262638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37645 (local address 127.0.0.1:46430)
0619 16:38:08.262802 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.262816 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.262885 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.263167 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.263169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.263330 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.263461 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.263467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.264478 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.264481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.265153 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.265160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.265783 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.265788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.265885 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.266287 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.266300 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.267407 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.269371 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.269374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.269376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.269602 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.269882 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.269887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.269888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.269969 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.270393 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.270397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.270488 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.270588 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.270704 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 16:38:08.270905 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.262760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.98.61:37645)
0619 16:38:08.262875 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.262879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.262891 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:38:08.262984 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.262990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.263052 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.263162 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.263583 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.264378 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.265259 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.265656 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.265778 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.267546 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.267553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.267557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.267590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.269229 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.269710 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.269718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.269720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.269792 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.270066 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.270069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.270071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.270284 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.270392 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.270657 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.270767 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:38:08.271756 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:38:08.272058 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.263215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.0.98.62:32993)
0619 16:38:08.263331 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.263335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.263347 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:38:08.263422 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.263427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.263466 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.263547 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.264217 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.264853 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.265807 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.266192 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.266297 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.268693 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.268697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.268701 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.268729 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.270220 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.270655 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.270659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.270660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.270743 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.271058 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.271062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.271064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.271271 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.271488 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.271808 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.271916 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:38:08.274493 13768 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } attempt: 3
I20260619 16:38:08.275529 13843 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } attempt: 3
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:38:08 dist-test-slave-0f28 krb5kdc[13687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:38:08.281635 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.272176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59330)
0619 16:38:08.272271 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.272281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.272337 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.272852 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.272855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.273032 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.273124 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.273131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.274054 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.274061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.274813 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.274822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.275451 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.275454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.275813 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.276225 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.276239 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.277717 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.279497 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.279501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.279503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.279728 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.280506 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.280510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.280512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.280610 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.281032 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.281036 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.281129 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.281257 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.281401 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":23}
I20260619 16:38:08.281641 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.272546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.0.98.60:38333)
0619 16:38:08.272652 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.272656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.272668 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:38:08.272699 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.272701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.272736 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.272813 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.273247 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.273943 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.274926 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.275266 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.275358 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.277829 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.277832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.277834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.277855 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.279397 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.280319 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.280323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.280325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.280399 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.280725 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.280729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.280731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.280917 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.281011 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.281291 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.281471 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":43}
I20260619 16:38:08.285658 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: Bootstrap starting.
I20260619 16:38:08.285949 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.275874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59358)
0619 16:38:08.276267 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.276279 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.276379 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.276610 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.276613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.276787 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.276863 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.276868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.278103 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.278108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.279013 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.279024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.279683 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.279689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.279797 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.280180 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.280199 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.281790 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.284430 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.284435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.284437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.284689 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.284951 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.284954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.284956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.285057 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.285507 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.285511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.285604 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.285691 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.285823 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"spinlock_wait_cycles":4992}
I20260619 16:38:08.286206 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.273412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.98.61:37645)
0619 16:38:08.273516 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.273520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.273533 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:38:08.273622 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.273625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.273661 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.273732 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.274294 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.274821 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.276494 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.276862 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.276958 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.282283 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.282289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.282294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.282322 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.284036 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.284483 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.284488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.284491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.284567 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.284867 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.284870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.284872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.285069 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.285264 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.285976 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.286080 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 16:38:08.286595 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.275929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.0.98.60:38333)
0619 16:38:08.276192 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.276206 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.276346 (+ 140us) server_negotiation.cc:407] Connection header received
0619 16:38:08.276450 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.276453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.276493 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.276961 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.277096 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.277941 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.279164 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.279543 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.279997 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.281949 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.281955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.281958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.281988 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.284328 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.284775 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.284778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.284780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.284858 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.285153 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.285157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.285158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.285379 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.286231 (+ 852us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.286319 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.286424 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260619 16:38:08.288252 13919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:08.288952 13919 log.cc:824] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:08.289100 13843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: Bootstrap starting.
I20260619 16:38:08.291018 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: No bootstrap required, opened a new log
I20260619 16:38:08.291611 13843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:08.291659 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.274822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38333 (local address 127.0.0.1:59344)
0619 16:38:08.274976 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:08.274989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:08.275054 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:08.275485 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:08.275488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:08.275667 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:08.275762 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.275768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.277752 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.277758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.278592 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:08.278603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.279309 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.279314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:08.279405 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.279834 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:08.279847 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:08.281651 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:08.290049 (+ 8398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.290053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.290055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.290303 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.290612 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:08.290615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:08.290617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:08.290712 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:08.291142 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:08.291145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:08.291241 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 16:38:08.291320 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 16:38:08.291478 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 16:38:08.292204 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.274905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.0.98.60:38333)
0619 16:38:08.275266 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.275274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.275286 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:38:08.275318 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.275323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.275362 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.275445 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.276084 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.277616 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.278756 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.279168 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.279605 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.281762 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.281768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.281770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.281792 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.289928 (+ 8136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.290412 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.290415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.290417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.290503 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.290812 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.290816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.290818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.291018 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.291828 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.291904 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.292025 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":300,"thread_start_us":98,"threads_started":1}
I20260619 16:38:08.294445 13768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: Bootstrap starting.
I20260619 16:38:08.294998 13919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.295210 13919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:08.295287 13919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94cdb0d302f942b7b9ae8786db0d1be2, State: Initialized, Role: FOLLOWER
I20260619 16:38:08.295677 13919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.296099 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.296221 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.296598 13768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:08.296854 13919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:08.297282 13768 log.cc:824] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:08.298343 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:38:08.299316 13768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: No bootstrap required, opened a new log
I20260619 16:38:08.301571 13843 log.cc:824] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: Log is configured to *not* fsync() on all Append() calls
W20260619 16:38:08.301952 13939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:38:08.302073 13939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:38:08.303490 13843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: No bootstrap required, opened a new log
I20260619 16:38:08.303437 13768 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.303669 13768 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:08.303728 13768 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcaccaeeef884b0a8ff0c1387c916270, State: Initialized, Role: FOLLOWER
I20260619 16:38:08.304191 13768 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.300936 13851 master_runner.cc:186] Error getting master registration for 127.0.98.62:32993: OK, instance_id { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" instance_seqno: 1781887087812315 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:38:08.304406 13851 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:38:08.305435 13768 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:08.307976 13843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.308241 13843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:08.308630 13843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53beed7eece34ebb8391276b97252e56, State: Initialized, Role: FOLLOWER
W20260619 16:38:08.309073 13721 tablet.cc:2369] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:08.309190 13843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
W20260619 16:38:08.310279 13960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:38:08.310355 13960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:38:08.311143 13843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:08.313592 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.302602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.0.98.61:37645)
0619 16:38:08.302703 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.302707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.304323 (+ 1616us) server_negotiation.cc:407] Connection header received
0619 16:38:08.304515 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.304521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.304569 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.304634 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.305618 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.306337 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.307502 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.307827 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.307906 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.309723 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.309727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.309729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.309754 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.311408 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.312054 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.312058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.312060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.312142 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.312442 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.312445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.312447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.312653 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.312751 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.313300 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.313446 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:38:08.310415 13700 master_runner.cc:186] Error getting master registration for 127.0.98.61:37645: OK, instance_id { permanent_uuid: "53beed7eece34ebb8391276b97252e56" instance_seqno: 1781887088024155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:38:08.314534 13700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:38:08.320144 13775 master_runner.cc:263] No leader master found from master 53beed7eece34ebb8391276b97252e56
I20260619 16:38:08.320209 13775 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:38:08.321506 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.321648 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.321846 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.321960 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.324156 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.303034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.0.98.62:32993)
0619 16:38:08.303141 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.303145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.314191 (+ 11046us) server_negotiation.cc:407] Connection header received
0619 16:38:08.314351 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.314354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.314396 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.314447 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.315304 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.316059 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.317156 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.317517 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.317594 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.320382 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.320386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.320389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.320413 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.322160 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.322660 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.322664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.322666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.322748 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.323058 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.323061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.323063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.323408 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.323511 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.323874 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.323995 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 16:38:08.324550 13973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:38:08.324620 13973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:38:08.332654 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:08.303672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.0.98.60:38333)
0619 16:38:08.303777 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:08.303780 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:08.314785 (+ 11005us) server_negotiation.cc:407] Connection header received
0619 16:38:08.314964 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:08.314969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:08.315023 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:08.315078 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:08.325415 (+ 10337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.325954 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.326778 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:08.327138 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:08.327238 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:08.328878 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:08.328882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:08.328884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:08.328913 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:08.330517 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.330990 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.330994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.330996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.331073 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:08.331344 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:08.331347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:08.331349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:08.331543 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:08.331689 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:08.332327 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 16:38:08.332464 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 16:38:08.482515 13872 tablet.cc:2369] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:38:08.520490 13796 tablet.cc:2369] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:08.665186 13962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:08.665432 13962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.666322 13962 leader_election.cc:288] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcaccaeeef884b0a8ff0c1387c916270 (127.0.98.62:32993), 94cdb0d302f942b7b9ae8786db0d1be2 (127.0.98.60:38333)
I20260619 16:38:08.666826 13893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53beed7eece34ebb8391276b97252e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" is_pre_election: true
I20260619 16:38:08.666821 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53beed7eece34ebb8391276b97252e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcaccaeeef884b0a8ff0c1387c916270" is_pre_election: true
I20260619 16:38:08.667137 13893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53beed7eece34ebb8391276b97252e56 in term 0.
I20260619 16:38:08.667160 13742 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53beed7eece34ebb8391276b97252e56 in term 0.
I20260619 16:38:08.667568 13791 leader_election.cc:302] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 53beed7eece34ebb8391276b97252e56, 94cdb0d302f942b7b9ae8786db0d1be2, fcaccaeeef884b0a8ff0c1387c916270; no voters:
I20260619 16:38:08.667842 13962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:38:08.667915 13962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:38:08.667945 13962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:08.669009 13962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.669385 13962 leader_election.cc:288] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [CANDIDATE]: Term 1 election: Requested vote from peers fcaccaeeef884b0a8ff0c1387c916270 (127.0.98.62:32993), 94cdb0d302f942b7b9ae8786db0d1be2 (127.0.98.60:38333)
I20260619 16:38:08.669713 13893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53beed7eece34ebb8391276b97252e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94cdb0d302f942b7b9ae8786db0d1be2"
I20260619 16:38:08.669729 13742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53beed7eece34ebb8391276b97252e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcaccaeeef884b0a8ff0c1387c916270"
I20260619 16:38:08.669809 13893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:08.669868 13742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:08.670799 13893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53beed7eece34ebb8391276b97252e56 in term 1.
I20260619 16:38:08.670928 13742 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53beed7eece34ebb8391276b97252e56 in term 1.
I20260619 16:38:08.671067 13791 leader_election.cc:302] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53beed7eece34ebb8391276b97252e56, 94cdb0d302f942b7b9ae8786db0d1be2; no voters:
I20260619 16:38:08.671238 13962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:08.671411 13962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [term 1 LEADER]: Becoming Leader. State: Replica: 53beed7eece34ebb8391276b97252e56, State: Running, Role: LEADER
I20260619 16:38:08.671716 13962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } }
I20260619 16:38:08.673527 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53beed7eece34ebb8391276b97252e56. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.673645 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:08.673942 13977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:08.674787 13977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:08.679030 13742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [term 1 FOLLOWER]: Refusing update from remote peer 53beed7eece34ebb8391276b97252e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:08.679195 13893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [term 1 FOLLOWER]: Refusing update from remote peer 53beed7eece34ebb8391276b97252e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:08.679700 13962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:08.679845 13975 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:08.683534 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53beed7eece34ebb8391276b97252e56. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.683550 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53beed7eece34ebb8391276b97252e56. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.683667 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.683665 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.685863 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.685988 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:08.686106 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.686213 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:08.686461 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.686534 13980 mvcc.cc:205] Tried to move back new op lower bound from 7298609515223560192 to 7298609515201724416. Current Snapshot: MvccSnapshot[applied={T|T < 7298609515223560192}]
I20260619 16:38:08.686573 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.687176 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53beed7eece34ebb8391276b97252e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcaccaeeef884b0a8ff0c1387c916270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32993 } } peers { permanent_uuid: "53beed7eece34ebb8391276b97252e56" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37645 } } peers { permanent_uuid: "94cdb0d302f942b7b9ae8786db0d1be2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38333 } } }
I20260619 16:38:08.687285 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:38:08.687397 13978 mvcc.cc:205] Tried to move back new op lower bound from 7298609515223560192 to 7298609515201724416. Current Snapshot: MvccSnapshot[applied={T|T < 7298609515223560192}]
I20260619 16:38:08.688086 13977 catalog_manager.cc:1347] Generated new cluster ID: 6fa6efa583e446c1bd0047ebd9210b63
I20260619 16:38:08.688143 13977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:08.698715 13977 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:38:08.699648 13977 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:08.704860 13977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53beed7eece34ebb8391276b97252e56: Generated new TSK 0
I20260619 16:38:08.705633 13977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:38:09.303467 13939 catalog_manager.cc:1259] Loaded cluster ID: 6fa6efa583e446c1bd0047ebd9210b63
I20260619 16:38:09.303557 13939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: loading cluster ID for follower catalog manager: success
I20260619 16:38:09.305553 13939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: acquiring CA information for follower catalog manager: success
I20260619 16:38:09.307049 13939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94cdb0d302f942b7b9ae8786db0d1be2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:38:09.311550 13960 catalog_manager.cc:1259] Loaded cluster ID: 6fa6efa583e446c1bd0047ebd9210b63
I20260619 16:38:09.311621 13960 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: loading cluster ID for follower catalog manager: success
I20260619 16:38:09.313424 13960 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: acquiring CA information for follower catalog manager: success
I20260619 16:38:09.314540 13960 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcaccaeeef884b0a8ff0c1387c916270: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:38:09.765357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13700
I20260619 16:38:09.772370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13775
I20260619 16:38:09.778290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 13851
2026-06-19T16:38:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3313 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:38:09 dist-test-slave-0f28 krb5kdc[13993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:38:09 dist-test-slave-0f28 krb5kdc[13993](info): set up 2 sockets
Jun 19 16:38:09 dist-test-slave-0f28 krb5kdc[13993](info): commencing operation
krb5kdc: starting...
W20260619 16:38:10.821907 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:38:10 dist-test-slave-0f28 krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:38:10.945895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:11.053692 14006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:11.053948 14006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:11.054037 14006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:11.057910 14006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:11.057993 14006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:11.058036 14006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:11.058068 14006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:11.058099 14006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:11.063097 14006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:11.064051 14006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:11.065035 14006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:11.073357 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:11.073364 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:11.073401 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:11.073562 14006 server_base.cc:1034] running on GCE node
I20260619 16:38:11.074172 14006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:11.075214 14006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:11.076391 14006 hybrid_clock.cc:648] HybridClock initialized: now 1781887091076385 us; error 35 us; skew 500 ppm
Jun 19 16:38:11 dist-test-slave-0f28 krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:11.079095 14006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:11.079855 14006 webserver.cc:460] Webserver started at http://127.0.98.62:44683/ using document root <none> and password file <none>
I20260619 16:38:11.080358 14006 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:11.080416 14006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:11.080572 14006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:11.082571 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "7bd01ba6f1b744e5bcabf57a1c82b107"
format_stamp: "Formatted at 2026-06-19 16:38:11 on dist-test-slave-0f28"
server_key: "6db5a6546a9ccaa5b085b560f19f9e12"
server_key_iv: "351e85d66471d12ee61e0a8792f853ad"
server_key_version: "encryptionkey@0"
I20260619 16:38:11.083115 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bd01ba6f1b744e5bcabf57a1c82b107"
format_stamp: "Formatted at 2026-06-19 16:38:11 on dist-test-slave-0f28"
server_key: "6db5a6546a9ccaa5b085b560f19f9e12"
server_key_iv: "351e85d66471d12ee61e0a8792f853ad"
server_key_version: "encryptionkey@0"
I20260619 16:38:11.086674 14006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:38:11.088945 14021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:11.090250 14006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:11.090373 14006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "7bd01ba6f1b744e5bcabf57a1c82b107"
format_stamp: "Formatted at 2026-06-19 16:38:11 on dist-test-slave-0f28"
server_key: "6db5a6546a9ccaa5b085b560f19f9e12"
server_key_iv: "351e85d66471d12ee61e0a8792f853ad"
server_key_version: "encryptionkey@0"
I20260619 16:38:11.090479 14006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:11.118382 14006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:11.121201 14006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:11.121424 14006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:11.128275 14006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41177
I20260619 16:38:11.128289 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41177 every 8 connection(s)
I20260619 16:38:11.129218 14006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:11.131487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14006
I20260619 16:38:11.131636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:38:11.131922 392 external_mini_cluster.cc:1428] Setting key 479f8c7e40b6e08f9aaf9f4adbb5b438
I20260619 16:38:11.131999 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:11.138236 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107: Bootstrap starting.
I20260619 16:38:11.140488 14074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107: Neither blocks nor log segments found. Creating new log.
Jun 19 16:38:11 dist-test-slave-0f28 krb5kdc[13993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:11.141242 14074 log.cc:824] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:11.142982 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107: No bootstrap required, opened a new log
I20260619 16:38:11.146070 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:11.133300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.98.62:41177)
0619 16:38:11.133927 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:11.133944 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:11.133981 (+ 37us) server_negotiation.cc:407] Connection header received
0619 16:38:11.134634 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:11.134651 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:11.134995 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:11.135360 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:11.136235 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:11.137305 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:11.138291 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:11.138855 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:11.139045 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:11.141615 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:11.141620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:11.141632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:11.141658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:11.143357 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:11.143810 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:11.143815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:11.143820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:11.143924 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:11.144204 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:11.144208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:11.144210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:11.144593 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:11.144797 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:11.145058 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 16:38:11.145388 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":124,"threads_started":1}
I20260619 16:38:11.146872 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } }
I20260619 16:38:11.147123 14074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:11.147204 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd01ba6f1b744e5bcabf57a1c82b107, State: Initialized, Role: FOLLOWER
I20260619 16:38:11.147670 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } }
I20260619 16:38:11.147877 14074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:11.147969 14074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:11.148085 14074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:11.148741 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:38:11.149077 14074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } }
I20260619 16:38:11.149454 14074 leader_election.cc:302] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bd01ba6f1b744e5bcabf57a1c82b107; no voters:
I20260619 16:38:11.149789 14074 leader_election.cc:288] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:38:11.149920 14079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:11.150131 14079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [term 1 LEADER]: Becoming Leader. State: Replica: 7bd01ba6f1b744e5bcabf57a1c82b107, State: Running, Role: LEADER
I20260619 16:38:11.150506 14079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } }
I20260619 16:38:11.151014 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:11.152005 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } }
I20260619 16:38:11.152123 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:11.152503 14088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:11.152477 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bd01ba6f1b744e5bcabf57a1c82b107. Latest consensus state: current_term: 1 leader_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bd01ba6f1b744e5bcabf57a1c82b107" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41177 } } }
I20260619 16:38:11.152597 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:11.155028 14088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:11.160679 14088 catalog_manager.cc:1347] Generated new cluster ID: dca6ac7954fa4cb1b03edf512b8ccfc8
I20260619 16:38:11.160742 14088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:11.175283 14088 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:38:11.176352 14088 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:11.180842 14088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bd01ba6f1b744e5bcabf57a1c82b107: Generated new TSK 0
I20260619 16:38:11.181541 14088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:38:11.204701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14006
2026-06-19T16:38:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1425 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:38:11 dist-test-slave-0f28 krb5kdc[14101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:38:11 dist-test-slave-0f28 krb5kdc[14101](info): set up 2 sockets
Jun 19 16:38:11 dist-test-slave-0f28 krb5kdc[14101](info): commencing operation
krb5kdc: starting...
W20260619 16:38:12.277535 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:38:12 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:38:12.396811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:12.500423 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:12.500684 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:12.500751 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:12.504609 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:12.504686 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:12.504724 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:12.504777 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:12.504817 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:12.509814 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:12.510758 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:12.511690 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:12.519585 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:12.519590 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:12.519618 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:12.519806 14114 server_base.cc:1034] running on GCE node
I20260619 16:38:12.520392 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:12.521518 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:12.522678 14114 hybrid_clock.cc:648] HybridClock initialized: now 1781887092522651 us; error 43 us; skew 500 ppm
Jun 19 16:38:12 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:12.530409 14114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:12.531170 14114 webserver.cc:460] Webserver started at http://127.0.98.62:39539/ using document root <none> and password file <none>
I20260619 16:38:12.531656 14114 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:12.531710 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:12.531858 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:12.533794 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "dfbbafc9e44041709acfa11b812ec629"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "150e6d20d31db2b6a463a7868241f7d7"
server_key_iv: "cd01d30963d94fa17839765f7a1c1b95"
server_key_version: "encryptionkey@0"
I20260619 16:38:12.534276 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "dfbbafc9e44041709acfa11b812ec629"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "150e6d20d31db2b6a463a7868241f7d7"
server_key_iv: "cd01d30963d94fa17839765f7a1c1b95"
server_key_version: "encryptionkey@0"
I20260619 16:38:12.537644 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:38:12.539839 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:12.541074 14114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:12.541203 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "dfbbafc9e44041709acfa11b812ec629"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "150e6d20d31db2b6a463a7868241f7d7"
server_key_iv: "cd01d30963d94fa17839765f7a1c1b95"
server_key_version: "encryptionkey@0"
I20260619 16:38:12.541303 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:12.554586 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:12.557391 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:12.557575 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:12.564250 14114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44357
I20260619 16:38:12.564520 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44357 every 8 connection(s)
I20260619 16:38:12.565218 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:12.567909 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:12.572113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14114
I20260619 16:38:12.572234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:38:12.572538 392 external_mini_cluster.cc:1428] Setting key 3f24470af937989c8e498daca86bddfd
I20260619 16:38:12.573566 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap starting.
I20260619 16:38:12.575798 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:12.576490 14182 log.cc:824] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:12.578433 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: No bootstrap required, opened a new log
Jun 19 16:38:12 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:12.581707 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:12.581936 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:12.582010 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbbafc9e44041709acfa11b812ec629, State: Initialized, Role: FOLLOWER
I20260619 16:38:12.582453 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:12.582592 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:12.582662 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:12.582754 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:12.583869 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:12.584208 14182 leader_election.cc:302] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbbafc9e44041709acfa11b812ec629; no voters:
I20260619 16:38:12.584539 14182 leader_election.cc:288] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:38:12.584650 14187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:12.584864 14187 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 LEADER]: Becoming Leader. State: Replica: dfbbafc9e44041709acfa11b812ec629, State: Running, Role: LEADER
I20260619 16:38:12.585243 14187 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:12.585777 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:12.586978 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbbafc9e44041709acfa11b812ec629. Latest consensus state: current_term: 1 leader_uuid: "dfbbafc9e44041709acfa11b812ec629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260619 16:38:12.586997 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfbbafc9e44041709acfa11b812ec629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260619 16:38:12.587177 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:12.587184 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:12.588167 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:12.573983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.98.62:44357)
0619 16:38:12.574411 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:12.574426 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:12.574462 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:38:12.575119 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:12.575137 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:12.575473 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:12.575811 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:12.576654 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:12.577976 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:12.578939 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:12.579435 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:12.579615 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:12.582290 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:12.582296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:12.582309 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:12.582335 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:12.584419 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:12.584991 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:12.584998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:12.585004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:12.585131 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:12.585531 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:12.585535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:12.585536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:12.586049 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:12.586280 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:12.586854 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 16:38:12.587153 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
W20260619 16:38:12.591691 14201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:38:12.591769 14201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:38:12.591856 14202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:12.592548 14202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:12.597653 14202 catalog_manager.cc:1347] Generated new cluster ID: be9630bd84e544bf85ba174f39b96424
I20260619 16:38:12.597725 14202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:12.618862 14202 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:38:12.619762 14202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:12.626474 14202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Generated new TSK 0
I20260619 16:38:12.627183 14202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:38:12.664286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44357
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:12.771914 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:12.772142 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:12.772183 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:12.776026 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:12.776075 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:12.776129 14208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:12.782377 14208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:12.783262 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:12.784157 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:12.793422 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:12.793503 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:12.793567 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:12.793643 14208 server_base.cc:1034] running on GCE node
I20260619 16:38:12.794085 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:12.794585 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:12.795786 14208 hybrid_clock.cc:648] HybridClock initialized: now 1781887092795772 us; error 28 us; skew 500 ppm
Jun 19 16:38:12 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:12.798502 14208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:12.799335 14208 webserver.cc:460] Webserver started at http://127.0.98.1:35527/ using document root <none> and password file <none>
I20260619 16:38:12.799846 14208 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:12.799928 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:12.800158 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:12.802103 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "04192b4fba22475fa0ae0d17c1a8127d"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "529e11bc3426d08af11b3f3e075b6a62"
server_key_iv: "98b14fff0016d86b9d47eb7478261f36"
server_key_version: "encryptionkey@0"
I20260619 16:38:12.802623 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "04192b4fba22475fa0ae0d17c1a8127d"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "529e11bc3426d08af11b3f3e075b6a62"
server_key_iv: "98b14fff0016d86b9d47eb7478261f36"
server_key_version: "encryptionkey@0"
I20260619 16:38:12.806104 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:38:12.808262 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:12.809572 14208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:12.809679 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "04192b4fba22475fa0ae0d17c1a8127d"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "529e11bc3426d08af11b3f3e075b6a62"
server_key_iv: "98b14fff0016d86b9d47eb7478261f36"
server_key_version: "encryptionkey@0"
I20260619 16:38:12.809770 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:12.819008 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:12.821673 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:12.821847 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:12.822369 14208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:12.823109 14208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:12.823153 14208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:12.823212 14208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:12.823244 14208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:12.831539 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43993
I20260619 16:38:12.831606 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43993 every 8 connection(s)
I20260619 16:38:12.832382 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:38:12.839705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14208
I20260619 16:38:12.839843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:38:12.840127 392 external_mini_cluster.cc:1428] Setting key 78b43b961e0cfaa0db3115142d714048
Jun 19 16:38:12 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:12.845250 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:12.834039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45213 (local address 127.0.98.62:44357)
0619 16:38:12.834165 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:12.834169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:12.835084 (+ 915us) server_negotiation.cc:407] Connection header received
0619 16:38:12.835887 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:12.835889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:12.835933 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:12.836005 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:12.837020 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:12.837592 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:12.838669 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:12.838895 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:12.838973 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:12.841492 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:12.841496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:12.841499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:12.841526 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:12.842922 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:12.843496 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:12.843499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:12.843500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:12.843567 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:12.843925 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:12.843927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:12.843928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:12.844101 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:12.844206 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:12.844794 (+ 588us) server_negotiation.cc:299] Negotiation successful
0619 16:38:12.845020 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:38:12.846107 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:12.834384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:45213)
0619 16:38:12.834969 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:12.835006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:12.835716 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:12.836150 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:12.836156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:12.836474 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:12.836905 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:12.836917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:12.837700 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:12.837703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:12.838558 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:12.838575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:12.839012 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:12.839014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:12.839142 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:12.839696 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:12.839710 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:12.841319 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:12.843017 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:12.843026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:12.843041 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:12.843402 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:12.843652 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:12.843655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:12.843657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:12.843833 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:12.844213 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:12.844222 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:12.844521 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 16:38:12.844719 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 16:38:12.845074 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":111,"threads_started":1}
I20260619 16:38:12.847273 14337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44357
I20260619 16:38:12.847496 14337 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:12.847947 14337 heartbeater.cc:511] Master 127.0.98.62:44357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:38:12.849429 14146 ts_manager.cc:194] Registered new tserver with Master: 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993)
I20260619 16:38:12.850577 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:38:12.869087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44357
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:12.975082 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:12.975314 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:12.975355 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:12.979269 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:12.979317 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:12.979405 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:12.984326 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:12.985246 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:12.986205 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:12.994967 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:12.995042 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:12.995019 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:12.995209 14342 server_base.cc:1034] running on GCE node
I20260619 16:38:12.995678 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:12.996212 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:12.997385 14342 hybrid_clock.cc:648] HybridClock initialized: now 1781887092997373 us; error 29 us; skew 500 ppm
Jun 19 16:38:12 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:13.000031 14342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:13.000782 14342 webserver.cc:460] Webserver started at http://127.0.98.2:39171/ using document root <none> and password file <none>
I20260619 16:38:13.001256 14342 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:13.001312 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:13.001529 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:13.003376 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "5666d60ffbf643b08c21d36b2888bb09"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "51c811c18f9db81259ef113bc4394d3a"
server_key_iv: "2b69a38198f455d78858be0fa16cdac3"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.003865 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5666d60ffbf643b08c21d36b2888bb09"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "51c811c18f9db81259ef113bc4394d3a"
server_key_iv: "2b69a38198f455d78858be0fa16cdac3"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.007342 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:38:13.009624 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.010759 14342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:38:13.010850 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "5666d60ffbf643b08c21d36b2888bb09"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "51c811c18f9db81259ef113bc4394d3a"
server_key_iv: "2b69a38198f455d78858be0fa16cdac3"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.010933 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:13.022828 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:13.025940 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:13.026081 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:13.026566 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:13.027283 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:13.027333 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.027374 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:13.027388 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.035574 14342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36193
I20260619 16:38:13.035622 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36193 every 8 connection(s)
I20260619 16:38:13.036456 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:38:13.044855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14342
I20260619 16:38:13.044987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:38:13.045256 392 external_mini_cluster.cc:1428] Setting key 7be23beba5b7923873c53b11ee136710
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:13.049834 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.038292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55819 (local address 127.0.98.62:44357)
0619 16:38:13.038420 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.038424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.039357 (+ 933us) server_negotiation.cc:407] Connection header received
0619 16:38:13.040206 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.040210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.040271 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.040370 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:13.041499 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.042019 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.043122 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.043339 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.043446 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.046108 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:13.046112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:13.046114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:13.046137 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:13.047620 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.048235 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.048241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.048243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.048340 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.048657 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.048662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.048665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.048893 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:13.049003 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.049631 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.049722 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:38:13.051146 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.038585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:55819)
0619 16:38:13.039197 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.039234 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.040002 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.040535 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.040542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.040889 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:13.041355 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.041370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.042148 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.042152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.042994 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.043003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.043483 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.043487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.043654 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.044218 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:13.044234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:13.045937 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:13.047733 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.047742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.047758 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.048124 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.048437 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.048440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.048441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.048561 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.049021 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:13.049027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:13.049311 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.049601 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.049975 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":112,"threads_started":1}
I20260619 16:38:13.052397 14471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44357
I20260619 16:38:13.052627 14471 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:13.053083 14471 heartbeater.cc:511] Master 127.0.98.62:44357 requested a full tablet report, sending...
I20260619 16:38:13.054167 14146 ts_manager.cc:194] Registered new tserver with Master: 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193)
I20260619 16:38:13.055052 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55819
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:38:13.073971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44357
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:13.180477 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:13.180718 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:13.180763 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:13.184751 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:13.184806 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:13.184893 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:13.190243 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:13.191233 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:13.192317 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:13.201395 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:13.201393 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:13.201565 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:13.201721 14476 server_base.cc:1034] running on GCE node
I20260619 16:38:13.202037 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:13.202561 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:13.203758 14476 hybrid_clock.cc:648] HybridClock initialized: now 1781887093203743 us; error 25 us; skew 500 ppm
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:13.206426 14476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:13.207278 14476 webserver.cc:460] Webserver started at http://127.0.98.3:40163/ using document root <none> and password file <none>
I20260619 16:38:13.207794 14476 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:13.207852 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:13.208078 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:13.210070 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6ab09de95d045fba62bc54cbcf7ca7c"
format_stamp: "Formatted at 2026-06-19 16:38:13 on dist-test-slave-0f28"
server_key: "3aa1cfa284ede3afe015db9954086d29"
server_key_iv: "5690982274b18718a869dc9d41f076b6"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.210597 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6ab09de95d045fba62bc54cbcf7ca7c"
format_stamp: "Formatted at 2026-06-19 16:38:13 on dist-test-slave-0f28"
server_key: "3aa1cfa284ede3afe015db9954086d29"
server_key_iv: "5690982274b18718a869dc9d41f076b6"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.214143 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:38:13.216338 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.217515 14476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:13.217631 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f6ab09de95d045fba62bc54cbcf7ca7c"
format_stamp: "Formatted at 2026-06-19 16:38:13 on dist-test-slave-0f28"
server_key: "3aa1cfa284ede3afe015db9954086d29"
server_key_iv: "5690982274b18718a869dc9d41f076b6"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.217725 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:13.232965 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:13.235685 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:13.235862 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:13.236377 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:13.237113 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:13.237177 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.237242 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:13.237283 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.246552 14476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36451
I20260619 16:38:13.246569 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36451 every 8 connection(s)
I20260619 16:38:13.247462 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:38:13.249372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14476
I20260619 16:38:13.249497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:38:13.249738 392 external_mini_cluster.cc:1428] Setting key 108be588aec7c985ca3ff1b37e224703
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:13.259924 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.249151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44185 (local address 127.0.98.62:44357)
0619 16:38:13.249298 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.249304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.250503 (+ 1199us) server_negotiation.cc:407] Connection header received
0619 16:38:13.251470 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.251473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.251529 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.251601 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:13.252627 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.253173 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.254299 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.254527 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.254626 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.256823 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:13.256826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:13.256828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:13.256851 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:13.258259 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.258662 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.258664 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.258666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.258735 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.259038 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.259041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.259042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.259226 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:13.259313 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.259719 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.259809 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:38:13.260617 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.249639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:44185)
0619 16:38:13.250357 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.250410 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.251286 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.251753 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.251759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.252078 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:13.252494 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.252505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.253289 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.253292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.254181 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.254190 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.254667 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.254670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.254823 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.255308 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:13.255323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:13.256681 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:13.258347 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.258351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.258361 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.258583 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.258808 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.258810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.258811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.258926 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.259326 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:13.259330 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:13.259522 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.259671 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.259878 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":139,"threads_started":1}
I20260619 16:38:13.261669 14605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44357
I20260619 16:38:13.261873 14605 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:13.262353 14605 heartbeater.cc:511] Master 127.0.98.62:44357 requested a full tablet report, sending...
I20260619 16:38:13.263314 14146 ts_manager.cc:194] Registered new tserver with Master: f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
I20260619 16:38:13.263847 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44185
I20260619 16:38:13.274782 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:38:13.283494 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.276597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.0.98.62:44357)
0619 16:38:13.276735 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.276739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.276883 (+ 144us) server_negotiation.cc:407] Connection header received
0619 16:38:13.276976 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.276979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.277015 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.277080 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:13.277785 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.278267 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.279017 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.279220 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.279296 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.280560 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:13.280563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:13.280565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:13.280586 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:13.281988 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.282414 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.282417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.282418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.282493 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.282711 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.282713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.282715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.282904 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:13.283003 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.283200 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.283285 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:38:13.286687 14146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:38:13.289987 14146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:38:13.303299 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.297538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43993 (local address 127.0.0.1:33292)
0619 16:38:13.297811 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.297853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.297961 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.298685 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.298692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.298714 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:13.298860 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.298870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.300416 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.300419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.301766 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.301773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.302777 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.302780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.302907 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.302935 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.303001 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.303142 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":225,"thread_start_us":66,"threads_started":1}
I20260619 16:38:13.303316 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.297692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.98.1:43993)
0619 16:38:13.298169 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.298185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.298209 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:38:13.298290 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.298295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.298501 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.299015 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:13.299182 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.300284 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.301976 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.302653 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.302900 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.303040 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.303103 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.303184 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":119,"threads_started":1}
I20260619 16:38:13.303599 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.297895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36451 (local address 127.0.0.1:40338)
0619 16:38:13.298405 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.298419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.298504 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.299515 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.299518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.299535 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:13.299643 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.299649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.301128 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.301131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.302215 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.302224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.303222 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.303226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.303304 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.303334 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.303388 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.303453 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260619 16:38:13.303759 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.298487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.0.98.2:36193)
0619 16:38:13.298750 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.298761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.298806 (+ 45us) server_negotiation.cc:407] Connection header received
0619 16:38:13.298930 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.298937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.299349 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.299490 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:13.299680 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.300411 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.301818 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.302578 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.302687 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.302819 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.303222 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.303300 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260619 16:38:13.304236 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.298292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.0.98.3:36451)
0619 16:38:13.299079 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.299090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.299110 (+ 20us) server_negotiation.cc:407] Connection header received
0619 16:38:13.299181 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.299186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.299357 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.299778 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:13.299917 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.300990 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.302333 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.303097 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.303863 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.303982 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.304026 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.304100 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":134,"threads_started":1}
I20260619 16:38:13.304406 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.298179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.0.1:40260)
0619 16:38:13.298728 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.298741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.298818 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.299468 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.299471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.299488 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:13.299585 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.299591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.300543 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.300546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.301665 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.301674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.302713 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.302718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.302824 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.302891 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.302990 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.303129 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":8,"thread_start_us":106,"threads_started":1}
I20260619 16:38:13.306253 14271 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 (DEFAULT_TABLE table=test-table [id=5607a7186a0a46e5a24cd5743c228ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:13.307178 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2093bcd5924db9b03d9ee6764c0081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:13.306855 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 (DEFAULT_TABLE table=test-table [id=5607a7186a0a46e5a24cd5743c228ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:13.306942 14405 tablet_service.cc:1467] Processing CreateTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 (DEFAULT_TABLE table=test-table [id=5607a7186a0a46e5a24cd5743c228ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:13.307825 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2093bcd5924db9b03d9ee6764c0081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:13.307853 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b2093bcd5924db9b03d9ee6764c0081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:13.312811 14622 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap starting.
I20260619 16:38:13.314689 14623 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Bootstrap starting.
I20260619 16:38:13.315402 14622 tablet_bootstrap.cc:654] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:13.315626 14624 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Bootstrap starting.
I20260619 16:38:13.316828 14623 tablet_bootstrap.cc:654] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:13.316816 14622 log.cc:824] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:13.317883 14624 tablet_bootstrap.cc:654] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:13.317987 14623 log.cc:824] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:13.318679 14624 log.cc:824] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:13.318819 14622 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: No bootstrap required, opened a new log
I20260619 16:38:13.318998 14622 ts_tablet_manager.cc:1397] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:38:13.319793 14623 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: No bootstrap required, opened a new log
I20260619 16:38:13.319967 14623 ts_tablet_manager.cc:1397] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:38:13.320390 14624 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: No bootstrap required, opened a new log
I20260619 16:38:13.320561 14624 ts_tablet_manager.cc:1397] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:38:13.322849 14622 raft_consensus.cc:348] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.323118 14622 raft_consensus.cc:374] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:13.323177 14622 raft_consensus.cc:729] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04192b4fba22475fa0ae0d17c1a8127d, State: Initialized, Role: FOLLOWER
I20260619 16:38:13.323594 14622 consensus_queue.cc:260] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.323588 14624 raft_consensus.cc:348] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.323616 14623 raft_consensus.cc:348] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.323822 14624 raft_consensus.cc:374] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:13.323841 14623 raft_consensus.cc:374] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:13.323872 14624 raft_consensus.cc:729] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5666d60ffbf643b08c21d36b2888bb09, State: Initialized, Role: FOLLOWER
I20260619 16:38:13.323892 14623 raft_consensus.cc:729] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ab09de95d045fba62bc54cbcf7ca7c, State: Initialized, Role: FOLLOWER
I20260619 16:38:13.324261 14624 consensus_queue.cc:260] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.324319 14623 consensus_queue.cc:260] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.324555 14622 ts_tablet_manager.cc:1428] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:38:13.324872 14337 heartbeater.cc:503] Master 127.0.98.62:44357 was elected leader, sending a full tablet report...
I20260619 16:38:13.324883 14471 heartbeater.cc:503] Master 127.0.98.62:44357 was elected leader, sending a full tablet report...
I20260619 16:38:13.325106 14624 ts_tablet_manager.cc:1428] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:38:13.325815 14623 ts_tablet_manager.cc:1428] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:38:13.325824 14605 heartbeater.cc:503] Master 127.0.98.62:44357 was elected leader, sending a full tablet report...
W20260619 16:38:13.333418 14338 tablet.cc:2369] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:13.334425 14630 raft_consensus.cc:482] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:13.334539 14630 raft_consensus.cc:504] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.335366 14630 leader_election.cc:288] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993), 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193)
I20260619 16:38:13.339222 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.335702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46647 (local address 127.0.98.1:43993)
0619 16:38:13.335815 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.335819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.335835 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:38:13.336058 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.336063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.336115 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.336190 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:13.336524 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.337145 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.338276 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.338688 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.338767 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.338844 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.339037 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.339107 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:38:13.339231 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.335635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43993 (local address 127.0.98.3:46647)
0619 16:38:13.335738 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.335755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.335835 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.336221 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.336223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.336238 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:13.336404 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.336411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.337242 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.337245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.338155 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.338165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.338805 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.338808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.338916 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.338939 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.339008 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.339095 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"spinlock_wait_cycles":25984}
I20260619 16:38:13.339263 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.335959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59819 (local address 127.0.98.2:36193)
0619 16:38:13.336085 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.336089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.336287 (+ 198us) server_negotiation.cc:407] Connection header received
0619 16:38:13.336402 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.336405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.336455 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.336537 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:13.336788 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.337281 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.338312 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.338709 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.338825 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.338891 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.339093 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.339163 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:38:13.339273 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.335895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.98.3:59819)
0619 16:38:13.336193 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.336206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.336324 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.336557 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.336560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.336578 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:13.336682 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.336690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.337411 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.337414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.338168 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.338176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.338842 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.338846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.338952 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.338969 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.339062 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.339133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"spinlock_wait_cycles":4096,"thread_start_us":115,"threads_started":1}
I20260619 16:38:13.340117 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2093bcd5924db9b03d9ee6764c0081" candidate_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5666d60ffbf643b08c21d36b2888bb09" is_pre_election: true
I20260619 16:38:13.340109 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2093bcd5924db9b03d9ee6764c0081" candidate_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04192b4fba22475fa0ae0d17c1a8127d" is_pre_election: true
I20260619 16:38:13.340369 14291 raft_consensus.cc:2454] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ab09de95d045fba62bc54cbcf7ca7c in term 0.
I20260619 16:38:13.340379 14425 raft_consensus.cc:2454] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6ab09de95d045fba62bc54cbcf7ca7c in term 0.
I20260619 16:38:13.340762 14492 leader_election.cc:302] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d, f6ab09de95d045fba62bc54cbcf7ca7c; no voters:
I20260619 16:38:13.341003 14630 raft_consensus.cc:2781] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:38:13.341101 14630 raft_consensus.cc:482] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:38:13.341159 14630 raft_consensus.cc:3037] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:13.342259 14630 raft_consensus.cc:504] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.342525 14630 leader_election.cc:288] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [CANDIDATE]: Term 1 election: Requested vote from peers 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993), 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193)
I20260619 16:38:13.342825 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2093bcd5924db9b03d9ee6764c0081" candidate_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04192b4fba22475fa0ae0d17c1a8127d"
I20260619 16:38:13.342909 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b2093bcd5924db9b03d9ee6764c0081" candidate_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5666d60ffbf643b08c21d36b2888bb09"
I20260619 16:38:13.342943 14291 raft_consensus.cc:3037] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:13.343061 14425 raft_consensus.cc:3037] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:13.343937 14291 raft_consensus.cc:2454] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ab09de95d045fba62bc54cbcf7ca7c in term 1.
I20260619 16:38:13.344110 14425 raft_consensus.cc:2454] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6ab09de95d045fba62bc54cbcf7ca7c in term 1.
I20260619 16:38:13.344185 14492 leader_election.cc:302] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d, f6ab09de95d045fba62bc54cbcf7ca7c; no voters:
I20260619 16:38:13.344367 14630 raft_consensus.cc:2781] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:13.344599 14630 raft_consensus.cc:686] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 LEADER]: Becoming Leader. State: Replica: f6ab09de95d045fba62bc54cbcf7ca7c, State: Running, Role: LEADER
I20260619 16:38:13.344915 14630 consensus_queue.cc:237] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.347817 14145 catalog_manager.cc:5495] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c reported cstate change: term changed from 0 to 1, leader changed from <none> to f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } health_report { overall_health: HEALTHY } } }
I20260619 16:38:13.358856 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.355709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.98.3:36451)
0619 16:38:13.355823 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.355827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.355838 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:38:13.355956 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.355958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.356002 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.356085 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:13.356268 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.356731 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.357605 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.357801 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.357878 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.358181 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:13.358521 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:38:13.358616 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.358669 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.358711 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:38:13.364190 14425 raft_consensus.cc:1261] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Refusing update from remote peer f6ab09de95d045fba62bc54cbcf7ca7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:13.364220 14291 raft_consensus.cc:1261] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Refusing update from remote peer f6ab09de95d045fba62bc54cbcf7ca7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:13.364903 14632 consensus_queue.cc:1035] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:13.365056 14630 consensus_queue.cc:1035] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:13.372375 14636 mvcc.cc:205] Tried to move back new op lower bound from 7298609534413234176 to 7298609534343446528. Current Snapshot: MvccSnapshot[applied={T|T < 7298609534413234176}]
I20260619 16:38:13.382047 14144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:13.382212 14144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:13.384316 14144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Sending DeleteTablet for 3 replicas of tablet 7b2093bcd5924db9b03d9ee6764c0081
I20260619 16:38:13.385052 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40338
I20260619 16:38:13.385052 14405 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40260
I20260619 16:38:13.385006 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33292
I20260619 16:38:13.385532 14647 tablet_replica.cc:330] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: stopping tablet replica
I20260619 16:38:13.385545 14646 tablet_replica.cc:330] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: stopping tablet replica
I20260619 16:38:13.385622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14208
I20260619 16:38:13.385886 14647 raft_consensus.cc:2229] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:13.386268 14647 raft_consensus.cc:2258] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:13.387693 14647 ts_tablet_manager.cc:1905] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:13.390453 14647 ts_tablet_manager.cc:1918] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:13.390587 14647 log.cc:1196] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/7b2093bcd5924db9b03d9ee6764c0081
I20260619 16:38:13.390738 14646 raft_consensus.cc:2229] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:38:13.391070 14647 ts_tablet_manager.cc:1939] T 7b2093bcd5924db9b03d9ee6764c0081 P 5666d60ffbf643b08c21d36b2888bb09: Deleting consensus metadata
I20260619 16:38:13.391129 14646 raft_consensus.cc:2258] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 16:38:13.392356 14130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43993 (error 108)
I20260619 16:38:13.392516 14646 ts_tablet_manager.cc:1905] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:13.392791 14130 catalog_manager.cc:4841] TS 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): tablet 7b2093bcd5924db9b03d9ee6764c0081 (table test-table [id=5607a7186a0a46e5a24cd5743c228ec7]) successfully deleted
I20260619 16:38:13.393429 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.393136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43993 (local address 127.0.0.1:33294)
0619 16:38:13.393259 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.393349 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 16:38:13.393579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14342
I20260619 16:38:13.396245 14646 ts_tablet_manager.cc:1918] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:13.396354 14646 log.cc:1196] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/7b2093bcd5924db9b03d9ee6764c0081
I20260619 16:38:13.396708 14646 ts_tablet_manager.cc:1939] T 7b2093bcd5924db9b03d9ee6764c0081 P f6ab09de95d045fba62bc54cbcf7ca7c: Deleting consensus metadata
W20260619 16:38:13.396834 14130 catalog_manager.cc:4584] TS 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b2093bcd5924db9b03d9ee6764c0081: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43993: connect: Connection refused (error 111)
I20260619 16:38:13.397899 14131 catalog_manager.cc:4841] TS f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): tablet 7b2093bcd5924db9b03d9ee6764c0081 (table test-table [id=5607a7186a0a46e5a24cd5743c228ec7]) successfully deleted
I20260619 16:38:13.401252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14476
I20260619 16:38:13.406629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14114
I20260619 16:38:13.412575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--webserver_interface=127.0.98.62
--webserver_port=39539
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:13.518602 14649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:13.518841 14649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:13.518877 14649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:13.522799 14649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:13.522847 14649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:13.522866 14649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:13.522882 14649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:13.522897 14649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:13.528080 14649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:13.528987 14649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:13.529971 14649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:13.542196 14655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:13.542196 14658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:13.542243 14656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:13.542459 14649 server_base.cc:1034] running on GCE node
I20260619 16:38:13.543015 14649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:13.544034 14649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:13.545178 14649 hybrid_clock.cc:648] HybridClock initialized: now 1781887093545147 us; error 45 us; skew 500 ppm
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:13.547748 14649 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:13.548537 14649 webserver.cc:460] Webserver started at http://127.0.98.62:39539/ using document root <none> and password file <none>
I20260619 16:38:13.549077 14649 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:13.549178 14649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:13.553073 14649 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 16:38:13.555150 14665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.556318 14649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:38:13.556447 14649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "dfbbafc9e44041709acfa11b812ec629"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "150e6d20d31db2b6a463a7868241f7d7"
server_key_iv: "cd01d30963d94fa17839765f7a1c1b95"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.557008 14649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:13.567145 14649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:13.569833 14649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:13.570019 14649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:13.577636 14649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44357
I20260619 16:38:13.577666 14717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44357 every 8 connection(s)
I20260619 16:38:13.578756 14649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:13.582994 14718 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:38:13.584555 14718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap starting.
I20260619 16:38:13.587939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14649
I20260619 16:38:13.588426 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44357
--webserver_port=35527
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:13.603175 14718 log.cc:824] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:13.611872 14718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:13.612262 14718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap complete.
I20260619 16:38:13.616494 14718 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:13.616873 14718 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbbafc9e44041709acfa11b812ec629, State: Initialized, Role: FOLLOWER
I20260619 16:38:13.617429 14718 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:13.617542 14718 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:13.617628 14718 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:13.617738 14718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:38:13.619660 14718 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:13.619982 14718 leader_election.cc:302] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbbafc9e44041709acfa11b812ec629; no voters:
I20260619 16:38:13.620287 14718 leader_election.cc:288] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:38:13.620366 14722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:38:13.620571 14722 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 LEADER]: Becoming Leader. State: Replica: dfbbafc9e44041709acfa11b812ec629, State: Running, Role: LEADER
I20260619 16:38:13.620843 14722 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:13.621485 14718 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:13.621760 14722 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbbafc9e44041709acfa11b812ec629. Latest consensus state: current_term: 2 leader_uuid: "dfbbafc9e44041709acfa11b812ec629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260619 16:38:13.621878 14722 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:13.622273 14723 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dfbbafc9e44041709acfa11b812ec629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260619 16:38:13.622434 14723 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:13.623497 14726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:13.628021 14726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:13.628587 14726 catalog_manager.cc:1259] Loaded cluster ID: be9630bd84e544bf85ba174f39b96424
I20260619 16:38:13.628646 14726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:13.629959 14726 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 16:38:13.634833 14726 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 16:38:13.639771 14726 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:13.640455 14726 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Loaded TSK: 0
I20260619 16:38:13.641610 14726 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:38:13.700354 14720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:13.700572 14720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:13.700613 14720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:13.704538 14720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:13.704587 14720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:13.704671 14720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:13.709618 14720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35527
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:13.710537 14720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:13.711440 14720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:13.721594 14748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:13.721751 14720 server_base.cc:1034] running on GCE node
W20260619 16:38:13.721589 14745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:13.721589 14746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:13.722328 14720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:13.722978 14720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:13.724160 14720 hybrid_clock.cc:648] HybridClock initialized: now 1781887093724138 us; error 42 us; skew 500 ppm
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:13.726980 14720 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:13.727833 14720 webserver.cc:460] Webserver started at http://127.0.98.1:35527/ using document root <none> and password file <none>
I20260619 16:38:13.728368 14720 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:13.728477 14720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:13.732197 14720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:13.734356 14755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.735540 14720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:13.735673 14720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "04192b4fba22475fa0ae0d17c1a8127d"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "529e11bc3426d08af11b3f3e075b6a62"
server_key_iv: "98b14fff0016d86b9d47eb7478261f36"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.736169 14720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:13.759398 14720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:13.762368 14720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:13.762562 14720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:13.763114 14720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:13.764124 14762 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:38:13.768141 14720 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:38:13.768210 14720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 16:38:13.768283 14720 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:38:13.769937 14720 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:38:13.770000 14720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:13.770241 14762 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap starting.
I20260619 16:38:13.780135 14720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43993
I20260619 16:38:13.780349 14869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43993 every 8 connection(s)
I20260619 16:38:13.781247 14720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:38:13.785362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14720
I20260619 16:38:13.785768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36193
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44357
--webserver_port=39171
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:13.789713 14762 log.cc:824] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:13.797195 14762 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:13.797643 14762 tablet_bootstrap.cc:492] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap complete.
I20260619 16:38:13.798967 14762 ts_tablet_manager.cc:1397] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent bootstrapping tablet: real 0.029s user 0.023s sys 0.004s
I20260619 16:38:13.800971 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.784812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57351 (local address 127.0.98.62:44357)
0619 16:38:13.785197 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.785215 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.785259 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:38:13.785598 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.785613 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.786252 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.786649 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:13.788122 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.789569 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.791397 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.792078 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.792247 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.795426 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:13.795433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:13.795447 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:13.795478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:13.797645 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.798277 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.798283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.798288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.798431 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.798802 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.798805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.798807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.799233 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:13.799349 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.799923 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.800212 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260619 16:38:13.801420 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.783657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:57351)
0619 16:38:13.784110 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.784159 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.785113 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.786736 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.786747 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.787253 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:13.787919 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.787940 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.789720 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.789726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.791257 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.791272 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.792231 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.792236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.792477 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.793169 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:13.793194 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:13.795218 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:13.797777 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.797782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.797794 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.798157 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.798536 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.798541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.798546 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.798716 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.799358 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:13.799364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:13.799622 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.799810 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.800150 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":142,"threads_started":1}
I20260619 16:38:13.801745 14762 raft_consensus.cc:348] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.802237 14762 raft_consensus.cc:729] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04192b4fba22475fa0ae0d17c1a8127d, State: Initialized, Role: FOLLOWER
I20260619 16:38:13.802680 14762 consensus_queue.cc:260] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } } peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } }
I20260619 16:38:13.803637 14762 ts_tablet_manager.cc:1428] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:38:13.803774 14870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44357
I20260619 16:38:13.803985 14870 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:13.804467 14870 heartbeater.cc:511] Master 127.0.98.62:44357 requested a full tablet report, sending...
I20260619 16:38:13.806967 14682 ts_manager.cc:194] Registered new tserver with Master: 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993)
I20260619 16:38:13.810738 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57351
I20260619 16:38:13.813184 14870 heartbeater.cc:503] Master 127.0.98.62:44357 was elected leader, sending a full tablet report...
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:38:13.819705 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.810056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43993 (local address 127.0.0.1:33300)
0619 16:38:13.810681 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.810712 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.810858 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.811369 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.811374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.811706 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:13.811799 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.811805 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.812756 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.812761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.813451 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.813461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.814148 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.814163 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.814271 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.814875 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:13.814889 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:13.816291 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:13.818062 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.818071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.818076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.818297 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.818588 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.818591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.818593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.818676 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.819090 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:13.819095 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:13.819214 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.819344 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.819521 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":117,"threads_started":1}
I20260619 16:38:13.819718 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.810102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.0.98.1:43993)
0619 16:38:13.810447 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.810458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.810799 (+ 341us) server_negotiation.cc:407] Connection header received
0619 16:38:13.811000 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.811004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.811236 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.811367 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:13.811914 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.812643 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.813563 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.814017 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.814138 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.816420 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:13.816428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:13.816445 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:13.816466 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:13.817934 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.818405 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.818410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.818412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.818489 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.818763 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.818767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.818769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.818980 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:13.819121 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.819431 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.819577 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":264,"thread_start_us":88,"threads_started":1}
I20260619 16:38:13.820640 14804 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b2093bcd5924db9b03d9ee6764c0081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33300
I20260619 16:38:13.821560 14879 tablet_replica.cc:330] stopping tablet replica
I20260619 16:38:13.821835 14879 raft_consensus.cc:2229] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:13.822070 14879 raft_consensus.cc:2258] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:13.823066 14879 ts_tablet_manager.cc:1905] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:13.827353 14879 ts_tablet_manager.cc:1918] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:13.827447 14879 log.cc:1196] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/7b2093bcd5924db9b03d9ee6764c0081
I20260619 16:38:13.827713 14879 ts_tablet_manager.cc:1939] T 7b2093bcd5924db9b03d9ee6764c0081 P 04192b4fba22475fa0ae0d17c1a8127d: Deleting consensus metadata
I20260619 16:38:13.829572 14666 catalog_manager.cc:4841] TS 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993): tablet 7b2093bcd5924db9b03d9ee6764c0081 (table test-table [id=5607a7186a0a46e5a24cd5743c228ec7]) successfully deleted
W20260619 16:38:13.903398 14874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:13.903621 14874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:13.903659 14874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:13.907469 14874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:13.907511 14874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:13.907564 14874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:13.912381 14874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39171
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:13.913295 14874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:13.914254 14874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:13.923498 14885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:13.923530 14887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:13.923617 14874 server_base.cc:1034] running on GCE node
W20260619 16:38:13.923556 14884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:13.924139 14874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:13.924711 14874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:13.925855 14874 hybrid_clock.cc:648] HybridClock initialized: now 1781887093925823 us; error 44 us; skew 500 ppm
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:13.928407 14874 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:13.929162 14874 webserver.cc:460] Webserver started at http://127.0.98.2:39171/ using document root <none> and password file <none>
I20260619 16:38:13.929713 14874 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:13.929802 14874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:13.933223 14874 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:13.935062 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.936141 14874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:13.936262 14874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "5666d60ffbf643b08c21d36b2888bb09"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "51c811c18f9db81259ef113bc4394d3a"
server_key_iv: "2b69a38198f455d78858be0fa16cdac3"
server_key_version: "encryptionkey@0"
I20260619 16:38:13.936772 14874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:13.970767 14874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:13.974049 14874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:13.974236 14874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:13.974793 14874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:13.975664 14874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:13.975728 14874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.975790 14874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:13.975833 14874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:13.984344 14874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36193
I20260619 16:38:13.984372 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36193 every 8 connection(s)
I20260619 16:38:13.985266 14874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:38:13 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:13.993813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 14874
I20260619 16:38:13.994199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36451
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44357
--webserver_port=40163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:13.997517 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.986982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57855 (local address 127.0.98.62:44357)
0619 16:38:13.987116 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:13.987120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:13.988099 (+ 979us) server_negotiation.cc:407] Connection header received
0619 16:38:13.988865 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:13.988867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:13.988914 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:13.988985 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:13.990073 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.990583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.991624 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.991838 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.991912 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.994190 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:13.994194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:13.994196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:13.994219 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:13.995589 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.996064 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.996068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.996069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.996146 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:13.996467 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:13.996470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:13.996472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:13.996645 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:13.996747 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:13.997199 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 16:38:13.997346 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:38:13.998230 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:13.987356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:57855)
0619 16:38:13.987951 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:13.987987 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:13.988694 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:13.989133 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:13.989139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:13.989490 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:13.989942 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.989955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.990681 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.990684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.991514 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:13.991522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:13.991950 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:13.991953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:13.992087 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:13.992577 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:13.992592 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:13.994034 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:13.995703 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.995709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.995721 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.995961 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.996233 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:13.996237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:13.996239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:13.996369 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:13.996745 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:13.996751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:13.996975 (+ 224us) client_negotiation.cc:769] Sending connection context
0619 16:38:13.997145 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 16:38:13.997469 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":105,"threads_started":1}
I20260619 16:38:13.999441 15008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44357
I20260619 16:38:13.999652 15008 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:14.000101 15008 heartbeater.cc:511] Master 127.0.98.62:44357 requested a full tablet report, sending...
I20260619 16:38:14.001092 14682 ts_manager.cc:194] Registered new tserver with Master: 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193)
I20260619 16:38:14.001686 14682 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57855
W20260619 16:38:14.100678 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:14.100982 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:14.101052 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:14.105014 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:14.105095 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:14.105182 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:14.110348 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40163
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:14.111426 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:14.112432 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:14.121917 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:14.121941 15019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:14.121974 15011 server_base.cc:1034] running on GCE node
W20260619 16:38:14.121922 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:14.122794 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:14.123436 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:14.124593 15011 hybrid_clock.cc:648] HybridClock initialized: now 1781887094124558 us; error 44 us; skew 500 ppm
Jun 19 16:38:14 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:14.127348 15011 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:14.128202 15011 webserver.cc:460] Webserver started at http://127.0.98.3:40163/ using document root <none> and password file <none>
I20260619 16:38:14.128755 15011 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:14.128854 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:14.132546 15011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:14.134531 15026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:14.135713 15011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:14.135846 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f6ab09de95d045fba62bc54cbcf7ca7c"
format_stamp: "Formatted at 2026-06-19 16:38:13 on dist-test-slave-0f28"
server_key: "3aa1cfa284ede3afe015db9954086d29"
server_key_iv: "5690982274b18718a869dc9d41f076b6"
server_key_version: "encryptionkey@0"
I20260619 16:38:14.136376 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:14.159245 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:14.162729 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:14.162938 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:14.163540 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:14.164335 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:14.164400 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:14.164494 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:14.164533 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:14.174227 15011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36451
I20260619 16:38:14.174266 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36451 every 8 connection(s)
I20260619 16:38:14.175227 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:38:14.180567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15011
Jun 19 16:38:14 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:14.188570 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.181231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.0.98.62:44357)
0619 16:38:14.181492 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.181497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.181597 (+ 100us) server_negotiation.cc:407] Connection header received
0619 16:38:14.181743 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.181746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.181788 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.181881 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:14.182562 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.183068 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.183892 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.184152 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.184254 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.185566 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:14.185569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:14.185576 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:14.185611 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:14.186981 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:14.187386 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:14.187390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:14.187392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:14.187470 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:14.187727 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:14.187732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:14.187733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:14.187898 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:14.188110 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.188248 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.188362 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":107,"threads_started":1}
I20260619 16:38:14.188746 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.177088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55563 (local address 127.0.98.62:44357)
0619 16:38:14.177230 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.177233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.178355 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 16:38:14.179231 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.179235 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.179292 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.179395 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:14.180610 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.181353 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.182497 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.182728 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.182834 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.185229 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:14.185232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:14.185234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:14.185254 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:14.186899 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:14.187366 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:14.187371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:14.187373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:14.187446 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:14.187778 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:14.187783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:14.187785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:14.187947 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:14.188042 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.188492 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.188602 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":55}
I20260619 16:38:14.189775 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.177560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:55563)
0619 16:38:14.178211 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.178250 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:14.179033 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:14.179561 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:14.179568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:14.179919 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:14.180445 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.180464 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.181502 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.181504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.182373 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.182382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.182868 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.182872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.183033 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.183614 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:14.183629 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:14.185074 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:14.187009 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:14.187015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:14.187027 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:14.187269 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:14.187538 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:14.187541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:14.187543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:14.187679 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:14.188053 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:14.188060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:14.188282 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 16:38:14.188451 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 16:38:14.188756 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":105,"threads_started":1}
I20260619 16:38:14.190726 15140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44357
I20260619 16:38:14.190944 15140 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:14.191388 15140 heartbeater.cc:511] Master 127.0.98.62:44357 requested a full tablet report, sending...
I20260619 16:38:14.192257 14679 ts_manager.cc:194] Registered new tserver with Master: f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
I20260619 16:38:14.192781 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55563
I20260619 16:38:14.201819 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:38:14.205585 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.202387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.0.98.62:44357)
0619 16:38:14.202504 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.202508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.202648 (+ 140us) server_negotiation.cc:407] Connection header received
0619 16:38:14.202775 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.202777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.202858 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.202930 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:14.203113 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.203565 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.204346 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.204545 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.204634 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.204862 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:14.205188 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:38:14.205270 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.205397 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.205437 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:38:14.206163 14679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:38:14.209296 14679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:38:14.216187 14804 tablet_service.cc:1467] Processing CreateTablet for tablet e2d1f4f1adfb48dbb877b662ac1225d8 (DEFAULT_TABLE table=test-table [id=26df1e8c3ca14443bc0a355aac19ad28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:14.216737 14804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d1f4f1adfb48dbb877b662ac1225d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:14.219412 14762 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap starting.
I20260619 16:38:14.220191 14762 tablet_bootstrap.cc:654] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:14.220418 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.215200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.0.1:40274)
0619 16:38:14.215304 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.215318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:14.215385 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:14.216469 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:14.216472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:14.216490 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:14.216588 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.216595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.217715 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.217718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.218878 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.218887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.219921 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.219924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.220031 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.220089 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 16:38:14.220185 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 16:38:14.220293 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 16:38:14.220465 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.215404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.0.98.2:36193)
0619 16:38:14.215984 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.215995 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.216017 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:38:14.216093 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.216097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.216292 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.216487 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:14.216706 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.217568 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.219002 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.219791 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.219915 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.220043 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.220228 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.220316 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":157,"threads_started":1}
I20260619 16:38:14.221735 14762 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: No bootstrap required, opened a new log
I20260619 16:38:14.221825 14762 ts_tablet_manager.cc:1397] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:14.222203 14762 raft_consensus.cc:348] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.222314 14762 raft_consensus.cc:374] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:14.222369 14762 raft_consensus.cc:729] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04192b4fba22475fa0ae0d17c1a8127d, State: Initialized, Role: FOLLOWER
I20260619 16:38:14.222507 14762 consensus_queue.cc:260] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.222793 14762 ts_tablet_manager.cc:1428] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:38:14.223388 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.216039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.0.98.3:36451)
0619 16:38:14.216365 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.216377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.218163 (+ 1786us) server_negotiation.cc:407] Connection header received
0619 16:38:14.218237 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.218242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.218381 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.218526 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:14.218748 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.219792 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.221921 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.222557 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.222691 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.222844 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.223166 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.223259 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":122,"threads_started":1}
I20260619 16:38:14.223742 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.215643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36451 (local address 127.0.0.1:40350)
0619 16:38:14.217956 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.217968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:14.218059 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:14.218506 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:14.218509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:14.218526 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:14.218623 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.218631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.220129 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.220132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.221767 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.221777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.222920 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.222924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.223014 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.223036 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:38:14.223537 (+ 501us) client_negotiation.cc:240] Negotiation successful
0619 16:38:14.223624 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2259,"thread_start_us":99,"threads_started":1}
I20260619 16:38:14.224733 14942 tablet_service.cc:1467] Processing CreateTablet for tablet e2d1f4f1adfb48dbb877b662ac1225d8 (DEFAULT_TABLE table=test-table [id=26df1e8c3ca14443bc0a355aac19ad28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:14.225816 14942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d1f4f1adfb48dbb877b662ac1225d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:14.226794 15074 tablet_service.cc:1467] Processing CreateTablet for tablet e2d1f4f1adfb48dbb877b662ac1225d8 (DEFAULT_TABLE table=test-table [id=26df1e8c3ca14443bc0a355aac19ad28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:14.227762 15074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2d1f4f1adfb48dbb877b662ac1225d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:14.234133 15153 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Bootstrap starting.
I20260619 16:38:14.235217 15154 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Bootstrap starting.
I20260619 16:38:14.237059 15153 tablet_bootstrap.cc:654] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:14.237380 15154 tablet_bootstrap.cc:654] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:14.238070 15153 log.cc:824] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:14.238577 15154 log.cc:824] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:14.240178 15153 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: No bootstrap required, opened a new log
I20260619 16:38:14.240355 15154 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: No bootstrap required, opened a new log
I20260619 16:38:14.240372 15153 ts_tablet_manager.cc:1397] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 16:38:14.240614 15154 ts_tablet_manager.cc:1397] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260619 16:38:14.243242 15153 raft_consensus.cc:348] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.243273 15154 raft_consensus.cc:348] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.243456 15153 raft_consensus.cc:374] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:14.243496 15154 raft_consensus.cc:374] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:14.243512 15153 raft_consensus.cc:729] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5666d60ffbf643b08c21d36b2888bb09, State: Initialized, Role: FOLLOWER
I20260619 16:38:14.243554 15154 raft_consensus.cc:729] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6ab09de95d045fba62bc54cbcf7ca7c, State: Initialized, Role: FOLLOWER
I20260619 16:38:14.243984 15154 consensus_queue.cc:260] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.244064 15153 consensus_queue.cc:260] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.244609 15140 heartbeater.cc:503] Master 127.0.98.62:44357 was elected leader, sending a full tablet report...
I20260619 16:38:14.244920 15154 ts_tablet_manager.cc:1428] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:38:14.244953 15153 ts_tablet_manager.cc:1428] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260619 16:38:14.245268 15008 heartbeater.cc:503] Master 127.0.98.62:44357 was elected leader, sending a full tablet report...
W20260619 16:38:14.282743 14871 tablet.cc:2369] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:14.391512 14876 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:14.391741 14876 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.392902 14876 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
I20260619 16:38:14.396677 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.393338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55237 (local address 127.0.98.2:36193)
0619 16:38:14.393481 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.393488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.393742 (+ 254us) server_negotiation.cc:407] Connection header received
0619 16:38:14.393883 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.393886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.393941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.394025 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:14.394244 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.394737 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.395740 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.396146 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.396253 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.396312 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.396487 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.396545 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 16:38:14.396703 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.393304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59947 (local address 127.0.98.3:36451)
0619 16:38:14.393475 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.393480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.393639 (+ 159us) server_negotiation.cc:407] Connection header received
0619 16:38:14.393832 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.393835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.393876 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.393956 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:14.394211 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.394697 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.395772 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.396174 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.396283 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.396340 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.396512 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.396574 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 16:38:14.396720 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.393160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.98.1:55237)
0619 16:38:14.393650 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.393662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:14.393785 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:14.394034 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:14.394037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:14.394053 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:14.394147 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.394152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.394840 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.394845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.395623 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.395631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.396257 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.396260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.396347 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.396388 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 16:38:14.396473 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:38:14.396540 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":56,"thread_start_us":98,"threads_started":1}
I20260619 16:38:14.396750 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.393160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36451 (local address 127.0.98.1:59947)
0619 16:38:14.393541 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.393559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:14.393703 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:14.393970 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:14.393972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:14.393989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:14.394096 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.394101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.394806 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.394810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.395645 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:14.395654 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.396290 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.396293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:14.396390 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.396415 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:38:14.396496 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 16:38:14.396572 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260619 16:38:14.397294 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1f4f1adfb48dbb877b662ac1225d8" candidate_uuid: "04192b4fba22475fa0ae0d17c1a8127d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" is_pre_election: true
I20260619 16:38:14.397416 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1f4f1adfb48dbb877b662ac1225d8" candidate_uuid: "04192b4fba22475fa0ae0d17c1a8127d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5666d60ffbf643b08c21d36b2888bb09" is_pre_election: true
I20260619 16:38:14.397641 15094 raft_consensus.cc:2454] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04192b4fba22475fa0ae0d17c1a8127d in term 0.
I20260619 16:38:14.397686 14962 raft_consensus.cc:2454] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04192b4fba22475fa0ae0d17c1a8127d in term 0.
I20260619 16:38:14.398087 14757 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d, f6ab09de95d045fba62bc54cbcf7ca7c; no voters:
I20260619 16:38:14.398341 14876 raft_consensus.cc:2781] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:38:14.398422 14876 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:38:14.398454 14876 raft_consensus.cc:3037] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:14.399438 14876 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.399801 14876 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 1 election: Requested vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
I20260619 16:38:14.400098 14962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1f4f1adfb48dbb877b662ac1225d8" candidate_uuid: "04192b4fba22475fa0ae0d17c1a8127d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5666d60ffbf643b08c21d36b2888bb09"
I20260619 16:38:14.400172 15094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2d1f4f1adfb48dbb877b662ac1225d8" candidate_uuid: "04192b4fba22475fa0ae0d17c1a8127d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c"
I20260619 16:38:14.400229 14962 raft_consensus.cc:3037] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:14.400300 15094 raft_consensus.cc:3037] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:14.401197 14962 raft_consensus.cc:2454] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04192b4fba22475fa0ae0d17c1a8127d in term 1.
I20260619 16:38:14.401319 15094 raft_consensus.cc:2454] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04192b4fba22475fa0ae0d17c1a8127d in term 1.
I20260619 16:38:14.401476 14756 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d, 5666d60ffbf643b08c21d36b2888bb09; no voters:
I20260619 16:38:14.401654 14876 raft_consensus.cc:2781] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:14.401804 14876 raft_consensus.cc:686] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 LEADER]: Becoming Leader. State: Replica: 04192b4fba22475fa0ae0d17c1a8127d, State: Running, Role: LEADER
I20260619 16:38:14.402050 14876 consensus_queue.cc:237] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.404785 14682 catalog_manager.cc:5495] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d reported cstate change: term changed from 0 to 1, leader changed from <none> to 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "04192b4fba22475fa0ae0d17c1a8127d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } health_report { overall_health: HEALTHY } } }
I20260619 16:38:14.415226 15164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.411830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.0.98.1:43993)
0619 16:38:14.412094 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:14.412100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:14.412111 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:38:14.412167 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:14.412169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:14.412215 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:14.412309 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:14.412485 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.412971 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.413889 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:14.414108 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:14.414196 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:14.414456 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:14.414842 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:38:14.414936 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:14.415047 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 16:38:14.415094 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":127,"threads_started":1}
I20260619 16:38:14.418715 15094 raft_consensus.cc:1261] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Refusing update from remote peer 04192b4fba22475fa0ae0d17c1a8127d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:14.418716 14962 raft_consensus.cc:1261] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Refusing update from remote peer 04192b4fba22475fa0ae0d17c1a8127d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:14.419420 15161 consensus_queue.cc:1035] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:14.419590 14876 consensus_queue.cc:1035] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 16:38:14.426065 15141 tablet.cc:2369] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:14.426743 15165 mvcc.cc:205] Tried to move back new op lower bound from 7298609538733654016 to 7298609538672975872. Current Snapshot: MvccSnapshot[applied={T|T < 7298609538733654016}]
I20260619 16:38:14.436226 14682 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:14.436394 14682 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:14.438828 14682 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Sending DeleteTablet for 3 replicas of tablet e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:14.439525 14942 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d1f4f1adfb48dbb877b662ac1225d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40274
I20260619 16:38:14.439635 14804 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d1f4f1adfb48dbb877b662ac1225d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33300
I20260619 16:38:14.439699 15074 tablet_service.cc:1514] Processing DeleteTablet for tablet e2d1f4f1adfb48dbb877b662ac1225d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40350
I20260619 16:38:14.440016 15175 tablet_replica.cc:330] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: stopping tablet replica
I20260619 16:38:14.440187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14720
I20260619 16:38:14.440387 15175 raft_consensus.cc:2229] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:14.440827 15175 raft_consensus.cc:2258] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:14.440887 15176 tablet_replica.cc:330] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: stopping tablet replica
I20260619 16:38:14.441177 15176 raft_consensus.cc:2229] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:14.441457 15176 raft_consensus.cc:2258] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:14.443014 15175 ts_tablet_manager.cc:1905] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:14.445036 15175 ts_tablet_manager.cc:1918] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:14.445159 15175 log.cc:1196] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:14.445499 15175 ts_tablet_manager.cc:1939] T e2d1f4f1adfb48dbb877b662ac1225d8 P 5666d60ffbf643b08c21d36b2888bb09: Deleting consensus metadata
W20260619 16:38:14.446762 14666 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43993 (error 108)
I20260619 16:38:14.446899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14874
I20260619 16:38:14.447206 14666 catalog_manager.cc:4841] TS 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): tablet e2d1f4f1adfb48dbb877b662ac1225d8 (table test-table [id=26df1e8c3ca14443bc0a355aac19ad28]) successfully deleted
I20260619 16:38:14.447253 15176 ts_tablet_manager.cc:1905] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:14.448222 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.447931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43993 (local address 127.0.0.1:56580)
0619 16:38:14.448084 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.448145 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 16:38:14.448459 14666 catalog_manager.cc:4584] TS 04192b4fba22475fa0ae0d17c1a8127d (127.0.98.1:43993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2d1f4f1adfb48dbb877b662ac1225d8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43993: connect: Connection refused (error 111)
I20260619 16:38:14.449750 15176 ts_tablet_manager.cc:1918] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:14.449832 15176 log.cc:1196] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:14.450143 15176 ts_tablet_manager.cc:1939] T e2d1f4f1adfb48dbb877b662ac1225d8 P f6ab09de95d045fba62bc54cbcf7ca7c: Deleting consensus metadata
I20260619 16:38:14.451085 14667 catalog_manager.cc:4841] TS f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): tablet e2d1f4f1adfb48dbb877b662ac1225d8 (table test-table [id=26df1e8c3ca14443bc0a355aac19ad28]) successfully deleted
I20260619 16:38:14.453099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15011
I20260619 16:38:14.458614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 14649
I20260619 16:38:14.464984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--webserver_interface=127.0.98.62
--webserver_port=39539
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:14.569715 15178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:14.569977 15178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:14.570053 15178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:14.574038 15178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:14.574116 15178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:14.574154 15178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:14.574205 15178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:14.574242 15178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:14.579385 15178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:14.580332 15178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:14.581288 15178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:14.589215 15183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:14.589288 15186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:14.589409 15178 server_base.cc:1034] running on GCE node
W20260619 16:38:14.589262 15184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:14.589970 15178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:14.591022 15178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:14.592195 15178 hybrid_clock.cc:648] HybridClock initialized: now 1781887094592167 us; error 41 us; skew 500 ppm
Jun 19 16:38:14 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:14.594774 15178 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:14.595527 15178 webserver.cc:460] Webserver started at http://127.0.98.62:39539/ using document root <none> and password file <none>
I20260619 16:38:14.596042 15178 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:14.596133 15178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:14.599660 15178 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:14.601604 15193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:14.602723 15178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:38:14.602849 15178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "dfbbafc9e44041709acfa11b812ec629"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "150e6d20d31db2b6a463a7868241f7d7"
server_key_iv: "cd01d30963d94fa17839765f7a1c1b95"
server_key_version: "encryptionkey@0"
I20260619 16:38:14.603349 15178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:14.619678 15178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:14.622323 15178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:14.622509 15178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:14.628803 15178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44357
I20260619 16:38:14.629091 15245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44357 every 8 connection(s)
I20260619 16:38:14.629805 15178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:14.630234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15178
I20260619 16:38:14.630710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44357
--webserver_port=35527
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:14.634761 15246 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:38:14.636296 15246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap starting.
I20260619 16:38:14.652894 15246 log.cc:824] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:14.663411 15246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:14.663766 15246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: Bootstrap complete.
I20260619 16:38:14.667685 15246 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:14.668033 15246 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfbbafc9e44041709acfa11b812ec629, State: Initialized, Role: FOLLOWER
I20260619 16:38:14.668519 15246 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:14.668637 15246 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:14.668716 15246 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:14.668840 15246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:38:14.670641 15246 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:14.670958 15246 leader_election.cc:302] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfbbafc9e44041709acfa11b812ec629; no voters:
I20260619 16:38:14.671240 15246 leader_election.cc:288] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:38:14.671419 15250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:38:14.671787 15250 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [term 3 LEADER]: Becoming Leader. State: Replica: dfbbafc9e44041709acfa11b812ec629, State: Running, Role: LEADER
I20260619 16:38:14.672190 15250 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } }
I20260619 16:38:14.672427 15246 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:14.673559 15251 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dfbbafc9e44041709acfa11b812ec629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260619 16:38:14.673679 15251 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:14.674290 15252 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfbbafc9e44041709acfa11b812ec629. Latest consensus state: current_term: 3 leader_uuid: "dfbbafc9e44041709acfa11b812ec629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfbbafc9e44041709acfa11b812ec629" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44357 } } }
I20260619 16:38:14.674388 15252 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:14.677054 15266 catalog_manager.cc:1259] Loaded cluster ID: be9630bd84e544bf85ba174f39b96424
I20260619 16:38:14.677129 15266 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: loading cluster ID for follower catalog manager: success
W20260619 16:38:14.677964 15266 catalog_manager.cc:1576] T 00000000000000000000000000000000 P dfbbafc9e44041709acfa11b812ec629: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260619 16:38:14.678042 15266 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260619 16:38:14.678114 15259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:14.680178 15259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:14.680620 15259 catalog_manager.cc:1259] Loaded cluster ID: be9630bd84e544bf85ba174f39b96424
I20260619 16:38:14.680687 15259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 16:38:14.681190 15259 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa60c3cc9ad google::LogMessage::Fail() at ??:0
@ 0x7fa60c3cf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa60c3cc4de google::LogMessage::Flush() at ??:0
@ 0x7fa60c3cda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa616d0bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa616d0cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa616d09579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa616d4d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa60e61e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa60e61ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa60e6203f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa60e60f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa60fc536db start_thread at ??:0
@ 0x7fa60aeca61f clone at ??:0
W20260619 16:38:14.766189 15248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:14.766402 15248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:14.766444 15248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:14.770434 15248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:14.770483 15248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:14.770560 15248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:14.775890 15248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35527
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:14.776921 15248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:14.777974 15248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:14.788581 15287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:14.788735 15248 server_base.cc:1034] running on GCE node
W20260619 16:38:14.788589 15288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:14.788599 15290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:14.789430 15248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:14.790088 15248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:14.791260 15248 hybrid_clock.cc:648] HybridClock initialized: now 1781887094791235 us; error 43 us; skew 500 ppm
Jun 19 16:38:14 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:14.794008 15248 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:14.794869 15248 webserver.cc:460] Webserver started at http://127.0.98.1:35527/ using document root <none> and password file <none>
I20260619 16:38:14.795406 15248 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:14.795497 15248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:14.799381 15248 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:38:14.801607 15297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:14.802948 15248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:38:14.803097 15248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "04192b4fba22475fa0ae0d17c1a8127d"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "529e11bc3426d08af11b3f3e075b6a62"
server_key_iv: "98b14fff0016d86b9d47eb7478261f36"
server_key_version: "encryptionkey@0"
I20260619 16:38:14.803643 15248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:14.834084 15248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:14.837241 15248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:14.837459 15248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:14.838042 15248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:14.839131 15304 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:38:14.844446 15248 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:38:14.844522 15248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 16:38:14.844625 15248 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:38:14.846253 15248 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:38:14.846318 15248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260619 16:38:14.846633 15304 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap starting.
I20260619 16:38:14.856647 15248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43993
I20260619 16:38:14.856861 15411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43993 every 8 connection(s)
I20260619 16:38:14.857677 15248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:38:14.867980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15248
I20260619 16:38:14.868116 15304 log.cc:824] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:14.868443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36193
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44357
--webserver_port=39171
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:14.878561 15304 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:14.878943 15304 tablet_bootstrap.cc:492] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Bootstrap complete.
I20260619 16:38:14.880321 15304 ts_tablet_manager.cc:1397] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.005s
I20260619 16:38:14.883425 15304 raft_consensus.cc:348] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.883963 15304 raft_consensus.cc:729] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04192b4fba22475fa0ae0d17c1a8127d, State: Initialized, Role: FOLLOWER
I20260619 16:38:14.884516 15304 consensus_queue.cc:260] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:14.885619 15304 ts_tablet_manager.cc:1428] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260619 16:38:14.902828 15414 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:38:14.859499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:57895)
0619 16:38:14.859961 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.860012 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:14.860786 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:14.902116 (+ 41330us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":129,"threads_started":1}
W20260619 16:38:14.903330 15412 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44357 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44357: Client connection negotiation failed: client connection to 127.0.98.62:44357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:38:14.904074 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.903813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:35457)
0619 16:38:14.903961 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.904025 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:38:14.904724 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:14.904555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:36717)
0619 16:38:14.904642 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:14.904681 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 16:38:14.904968 15412 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:38:14.982177 15416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:14.982409 15416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:14.982450 15416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:14.986371 15416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:14.986418 15416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:14.986519 15416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:14.991680 15416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39171
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:14.992643 15416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:14.993633 15416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:15.002812 15423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:15.002957 15425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:15.002981 15422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:15.003155 15416 server_base.cc:1034] running on GCE node
I20260619 16:38:15.003526 15416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:15.004132 15416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:15.005307 15416 hybrid_clock.cc:648] HybridClock initialized: now 1781887095005291 us; error 27 us; skew 500 ppm
Jun 19 16:38:15 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:15.008069 15416 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:15.008818 15416 webserver.cc:460] Webserver started at http://127.0.98.2:39171/ using document root <none> and password file <none>
I20260619 16:38:15.009303 15416 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:15.009419 15416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:15.012945 15416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:15.014931 15432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:15.016153 15416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:15.016276 15416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "5666d60ffbf643b08c21d36b2888bb09"
format_stamp: "Formatted at 2026-06-19 16:38:12 on dist-test-slave-0f28"
server_key: "51c811c18f9db81259ef113bc4394d3a"
server_key_iv: "2b69a38198f455d78858be0fa16cdac3"
server_key_version: "encryptionkey@0"
I20260619 16:38:15.016786 15416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:15.048075 15416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:15.051237 15416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:15.051435 15416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:15.052018 15416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:15.052810 15416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:15.052867 15416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:15.052969 15416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:15.053010 15416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:15.061554 15416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36193
I20260619 16:38:15.061584 15545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36193 every 8 connection(s)
I20260619 16:38:15.062544 15416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:38:15.065965 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.064517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:40695)
0619 16:38:15.065141 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.065240 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":103,"threads_started":1}
W20260619 16:38:15.066421 15546 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44357 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44357: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
I20260619 16:38:15.066942 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.066759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:42079)
0619 16:38:15.066853 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.066893 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 16:38:15.067476 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.067258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:54589)
0619 16:38:15.067364 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.067418 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 16:38:15.067543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15416
W20260619 16:38:15.067653 15546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:38:15.067950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36451
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44357
--webserver_port=40163
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:15.176133 15549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:15.176359 15549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:15.176430 15549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:15.180403 15549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:15.180452 15549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:15.180536 15549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:15.185668 15549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40163
--tserver_master_addrs=127.0.98.62:44357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:15.186600 15549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:15.187530 15549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:15.196549 15554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:15.196586 15557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:15.196580 15555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:15.196663 15549 server_base.cc:1034] running on GCE node
I20260619 16:38:15.197131 15549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:15.197791 15549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:15.198998 15549 hybrid_clock.cc:648] HybridClock initialized: now 1781887095198967 us; error 44 us; skew 500 ppm
Jun 19 16:38:15 dist-test-slave-0f28 krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:15.201632 15549 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:15.202366 15549 webserver.cc:460] Webserver started at http://127.0.98.3:40163/ using document root <none> and password file <none>
I20260619 16:38:15.202839 15549 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:15.202896 15549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:15.206352 15549 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:38:15.208289 15564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:15.209479 15549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 16:38:15.209583 15549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f6ab09de95d045fba62bc54cbcf7ca7c"
format_stamp: "Formatted at 2026-06-19 16:38:13 on dist-test-slave-0f28"
server_key: "3aa1cfa284ede3afe015db9954086d29"
server_key_iv: "5690982274b18718a869dc9d41f076b6"
server_key_version: "encryptionkey@0"
I20260619 16:38:15.210176 15549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:15.242623 15549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:15.245811 15549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:15.245994 15549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:15.246563 15549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:15.247316 15549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:15.247361 15549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:15.247424 15549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:15.247457 15549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:15.255798 15549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36451
I20260619 16:38:15.255834 15677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36451 every 8 connection(s)
I20260619 16:38:15.256727 15549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:38:15.260234 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.258673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:41623)
0619 16:38:15.259122 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.259589 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":108,"threads_started":1}
W20260619 16:38:15.260694 15678 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44357 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44357: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
I20260619 16:38:15.261262 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.261059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:47763)
0619 16:38:15.261158 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.261203 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 16:38:15.261799 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.261606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:54521)
0619 16:38:15.261703 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.261745 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 16:38:15.261974 15678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:38:15.264029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15549
I20260619 16:38:15.905975 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:15.905597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:48473)
0619 16:38:15.905820 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:15.905870 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260619 16:38:16.036960 15683 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:16.037173 15683 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:16.038321 15683 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
Jun 19 16:38:16 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 16:38:16 dist-test-slave-0f28 krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 16:38:16.054617 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.038541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36193 (local address 127.0.98.1:45297)
0619 16:38:16.038878 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:16.038894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:16.039003 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:16.040904 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:16.040910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:16.041535 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:16.042387 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:16.042408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.044309 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.044312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:16.045959 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:16.045970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.046874 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.046878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:16.047152 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:16.047872 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:16.047895 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:16.049831 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:16.051855 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:16.051861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:16.051874 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:16.052206 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:16.052540 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:16.052543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:16.052546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:16.052690 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:16.053637 (+ 947us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:16.053643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:16.053873 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 16:38:16.054047 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 16:38:16.054380 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":317,"spinlock_wait_cycles":19712}
I20260619 16:38:16.054634 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.038710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45297 (local address 127.0.98.2:36193)
0619 16:38:16.039105 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:16.039120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:16.039164 (+ 44us) server_negotiation.cc:407] Connection header received
0619 16:38:16.040014 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:16.040023 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:16.040475 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:16.040805 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:16.042595 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.044158 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.046072 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.046759 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.046919 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:16.050044 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:16.050049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:16.050061 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:16.050095 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:16.051741 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:16.052321 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:16.052325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:16.052329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:16.052433 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:16.052988 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:16.052994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:16.052999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:16.053520 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:16.053662 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:16.054203 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 16:38:16.054456 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":93,"threads_started":1}
I20260619 16:38:16.054659 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.038709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39385 (local address 127.0.98.3:36451)
0619 16:38:16.039105 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:16.039121 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:16.039164 (+ 43us) server_negotiation.cc:407] Connection header received
0619 16:38:16.040015 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:16.040023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:16.040476 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:16.040805 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:16.042679 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.044158 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.046072 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.046759 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.046919 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:16.050370 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:16.050378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:16.050394 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:16.050428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:16.052351 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:16.052751 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:16.052756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:16.052761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:16.052880 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:16.053381 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:16.053384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:16.053386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:16.053740 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:16.053876 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:16.054207 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 16:38:16.054467 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":90,"threads_started":1}
I20260619 16:38:16.054673 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.038541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36451 (local address 127.0.98.1:39385)
0619 16:38:16.038878 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:16.038894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:16.039021 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:16.040904 (+ 1883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:16.040910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:16.041535 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:16.042387 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:16.042408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.044310 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.044312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:16.045959 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:16.045970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:16.046874 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:16.046878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:16.047155 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:16.047870 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:16.047905 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:16.050180 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:16.052444 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:16.052447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:16.052451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:16.052646 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:16.052971 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:16.052975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:16.052977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:16.053057 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:16.053847 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:16.053850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:16.053950 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 16:38:16.054053 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 16:38:16.054467 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":621,"spinlock_wait_cycles":42624,"thread_start_us":104,"threads_started":1}
W20260619 16:38:16.057708 15299 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
W20260619 16:38:16.057854 15298 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:16.057931 15299 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d; no voters: 5666d60ffbf643b08c21d36b2888bb09, f6ab09de95d045fba62bc54cbcf7ca7c
I20260619 16:38:16.058176 15683 raft_consensus.cc:2726] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:38:16.068478 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.068135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:58187)
0619 16:38:16.068336 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:16.068393 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
I20260619 16:38:16.262815 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.262442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:43717)
0619 16:38:16.262676 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:16.262730 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260619 16:38:16.907266 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:16.906780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:33393)
0619 16:38:16.907088 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:16.907145 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
I20260619 16:38:17.069710 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:17.069117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:53185)
0619 16:38:17.069553 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:17.069611 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260619 16:38:17.263891 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:17.263497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:38779)
0619 16:38:17.263743 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:17.263795 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260619 16:38:17.723987 15692 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:17.724123 15692 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:17.724507 15692 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
W20260619 16:38:17.725270 15298 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
W20260619 16:38:17.725407 15299 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:17.725515 15299 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d; no voters: 5666d60ffbf643b08c21d36b2888bb09, f6ab09de95d045fba62bc54cbcf7ca7c
I20260619 16:38:17.725663 15692 raft_consensus.cc:2726] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:38:17.908586 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:17.908195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:42363)
0619 16:38:17.908432 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:17.908486 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260619 16:38:18.070742 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:18.070383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:58515)
0619 16:38:18.070609 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:18.070659 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260619 16:38:18.264928 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:18.264577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:50925)
0619 16:38:18.264795 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:18.264844 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
I20260619 16:38:18.909759 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:18.909296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:51071)
0619 16:38:18.909604 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:18.909662 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
I20260619 16:38:19.071789 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:19.071390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:60997)
0619 16:38:19.071639 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:19.071695 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260619 16:38:19.266011 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:19.265651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:58679)
0619 16:38:19.265877 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:19.265927 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260619 16:38:19.910921 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:19.910533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:36561)
0619 16:38:19.910779 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:19.910828 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":68,"threads_started":1}
I20260619 16:38:20.073001 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:20.072575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:47523)
0619 16:38:20.072858 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:20.072912 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260619 16:38:20.267160 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:20.266756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:54947)
0619 16:38:20.267016 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:20.267073 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
I20260619 16:38:20.912091 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:20.911650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:55955)
0619 16:38:20.911921 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:20.911978 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
I20260619 16:38:21.074178 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:21.073776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:41443)
0619 16:38:21.074035 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:21.074090 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260619 16:38:21.092983 15704 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:21.093169 15704 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:21.093668 15704 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
W20260619 16:38:21.094457 15299 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
W20260619 16:38:21.094543 15298 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:21.094668 15298 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d; no voters: 5666d60ffbf643b08c21d36b2888bb09, f6ab09de95d045fba62bc54cbcf7ca7c
I20260619 16:38:21.094832 15704 raft_consensus.cc:2726] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:38:21.268251 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:21.267881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:46443)
0619 16:38:21.268116 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:21.268167 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260619 16:38:21.913244 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:21.912809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:45869)
0619 16:38:21.913090 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:21.913148 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260619 16:38:22.075475 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:22.074938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:42353)
0619 16:38:22.075307 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:22.075384 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":73,"threads_started":1}
I20260619 16:38:22.269555 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:22.269056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:38789)
0619 16:38:22.269400 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:22.269457 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260619 16:38:22.914454 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:22.914055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:50071)
0619 16:38:22.914312 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:22.914363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
I20260619 16:38:23.076639 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:23.076244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:55211)
0619 16:38:23.076490 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:23.076544 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260619 16:38:23.270745 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:23.270327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:48617)
0619 16:38:23.270588 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:23.270646 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260619 16:38:23.915573 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:23.915134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:35127)
0619 16:38:23.915413 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:23.915477 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":72,"threads_started":1}
I20260619 16:38:24.077746 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:24.077295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:60243)
0619 16:38:24.077603 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:24.077656 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":80,"threads_started":1}
I20260619 16:38:24.271917 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:24.271513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:54409)
0619 16:38:24.271763 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:24.271813 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":85,"threads_started":1}
I20260619 16:38:24.916769 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:24.916386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:59403)
0619 16:38:24.916620 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:24.916675 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260619 16:38:25.079001 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:25.078554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:42395)
0619 16:38:25.078845 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:25.078906 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260619 16:38:25.273140 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:25.272735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:46967)
0619 16:38:25.272994 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:25.273048 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":92,"threads_started":1}
I20260619 16:38:25.457369 15718 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:25.457535 15718 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:25.457947 15718 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
W20260619 16:38:25.458714 15299 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
W20260619 16:38:25.458855 15298 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:25.458953 15298 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d; no voters: 5666d60ffbf643b08c21d36b2888bb09, f6ab09de95d045fba62bc54cbcf7ca7c
I20260619 16:38:25.459112 15718 raft_consensus.cc:2726] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:38:25.917948 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:25.917512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:59717)
0619 16:38:25.917753 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:25.917807 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260619 16:38:26.080210 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:26.079804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:43205)
0619 16:38:26.080067 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:26.080121 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260619 16:38:26.274319 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:26.273907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:44571)
0619 16:38:26.274164 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:26.274218 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260619 16:38:26.919097 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:26.918667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:37363)
0619 16:38:26.918947 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:26.919002 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":92,"threads_started":1}
I20260619 16:38:27.034052 15723 raft_consensus.cc:482] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:27.034231 15723 raft_consensus.cc:504] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5666d60ffbf643b08c21d36b2888bb09" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36193 } } peers { permanent_uuid: "f6ab09de95d045fba62bc54cbcf7ca7c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36451 } } peers { permanent_uuid: "04192b4fba22475fa0ae0d17c1a8127d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43993 } }
I20260619 16:38:27.034761 15723 leader_election.cc:288] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193), f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451)
W20260619 16:38:27.035593 15299 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f6ab09de95d045fba62bc54cbcf7ca7c (127.0.98.3:36451): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
W20260619 16:38:27.035699 15298 leader_election.cc:341] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5666d60ffbf643b08c21d36b2888bb09 (127.0.98.2:36193): Not found: Tablet not found: e2d1f4f1adfb48dbb877b662ac1225d8
I20260619 16:38:27.035807 15299 leader_election.cc:302] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04192b4fba22475fa0ae0d17c1a8127d; no voters: 5666d60ffbf643b08c21d36b2888bb09, f6ab09de95d045fba62bc54cbcf7ca7c
I20260619 16:38:27.036005 15723 raft_consensus.cc:2726] T e2d1f4f1adfb48dbb877b662ac1225d8 P 04192b4fba22475fa0ae0d17c1a8127d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:38:27.081287 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:27.080913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:35529)
0619 16:38:27.081140 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:27.081192 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":67,"threads_started":1}
I20260619 16:38:27.275475 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:27.275076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:38071)
0619 16:38:27.275331 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:27.275381 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260619 16:38:27.920450 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:27.920008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:33561)
0619 16:38:27.920269 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:27.920353 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":70,"threads_started":1}
I20260619 16:38:28.082557 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:28.082128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:55827)
0619 16:38:28.082386 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:28.082461 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260619 16:38:28.276738 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:28.276353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:33315)
0619 16:38:28.276592 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:28.276646 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260619 16:38:28.921619 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:28.921191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:45383)
0619 16:38:28.921465 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:28.921517 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260619 16:38:29.083709 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:29.083317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:46175)
0619 16:38:29.083563 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:29.083616 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260619 16:38:29.277918 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:29.277528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:60951)
0619 16:38:29.277767 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:29.277826 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260619 16:38:29.922885 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:29.922481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.1:48581)
0619 16:38:29.922737 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:29.922791 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":66,"threads_started":1}
I20260619 16:38:30.084910 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:30.084552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.2:36463)
0619 16:38:30.084769 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:30.084817 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260619 16:38:30.271210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15248
I20260619 16:38:30.276774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15416
I20260619 16:38:30.279078 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:30.278702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44357 (local address 127.0.98.3:33403)
0619 16:38:30.278938 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:30.278991 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260619 16:38:30.282158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15549
2026-06-19T16:38:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19079 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:38:30 dist-test-slave-0f28 krb5kdc[15737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:38:30 dist-test-slave-0f28 krb5kdc[15737](info): set up 2 sockets
Jun 19 16:38:30 dist-test-slave-0f28 krb5kdc[15737](info): commencing operation
krb5kdc: starting...
W20260619 16:38:31.325675 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:38:31 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:38:31.444373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:31.549235 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:31.549474 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:31.549518 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:31.553364 15750 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:38:31.553423 15750 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:38:31.553438 15750 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:38:31.553452 15750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:31.553468 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:31.553493 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:31.553508 15750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:31.553522 15750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:31.558384 15750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:31.559303 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:31.560256 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:38:31.568522 15750 server_base.cc:1034] running on GCE node
W20260619 16:38:31.568462 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:31.568460 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:31.568470 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:31.569420 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:31.570405 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:31.571633 15750 hybrid_clock.cc:648] HybridClock initialized: now 1781887111571616 us; error 31 us; skew 500 ppm
Jun 19 16:38:31 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:31.574592 15750 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:31.575659 15750 webserver.cc:460] Webserver started at http://127.0.98.62:42541/ using document root <none> and password file <none>
I20260619 16:38:31.576212 15750 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:31.576273 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:31.576479 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:31.578528 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "7ec1beef6b2d493d9183ce0ff0332260"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "d0937e9d610344bc3ac30647ab03a767"
server_key_iv: "6f5c23cb210050d8c853aa748558c113"
server_key_version: "encryptionkey@0"
I20260619 16:38:31.579111 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ec1beef6b2d493d9183ce0ff0332260"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "d0937e9d610344bc3ac30647ab03a767"
server_key_iv: "6f5c23cb210050d8c853aa748558c113"
server_key_version: "encryptionkey@0"
I20260619 16:38:31.582617 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:38:31.584827 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:31.585963 15750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:31.586112 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "7ec1beef6b2d493d9183ce0ff0332260"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "d0937e9d610344bc3ac30647ab03a767"
server_key_iv: "6f5c23cb210050d8c853aa748558c113"
server_key_version: "encryptionkey@0"
I20260619 16:38:31.586220 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:31.616283 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:31.619237 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:31.619448 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:31.626425 15750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36777
I20260619 16:38:31.626706 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36777 every 8 connection(s)
I20260619 16:38:31.627357 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:31.629714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15750
I20260619 16:38:31.629864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:38:31.630067 15818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:31.630227 392 external_mini_cluster.cc:1428] Setting key fab954b74b296e9610e92c6d81298d4d
I20260619 16:38:31.637370 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap starting.
Jun 19 16:38:31 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:31.639583 15818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:31.640519 15818 log.cc:824] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:31.642793 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: No bootstrap required, opened a new log
I20260619 16:38:31.644953 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:31.631571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.0.98.62:36777)
0619 16:38:31.631959 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:31.631978 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:31.632027 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:38:31.632862 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:31.632884 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:31.633289 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:31.633663 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:31.634288 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:31.635718 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:31.636603 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:31.637253 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:31.637451 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:31.639946 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:31.639952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:31.639967 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:31.640004 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:31.641892 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:31.642305 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:31.642310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:31.642316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:31.642443 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:31.642686 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:31.642689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:31.642691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:31.643168 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:31.643355 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:31.643587 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 16:38:31.643852 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260619 16:38:31.646597 15818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:31.646852 15818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:31.647003 15818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec1beef6b2d493d9183ce0ff0332260, State: Initialized, Role: FOLLOWER
I20260619 16:38:31.647388 15818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:31.647505 15818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:31.647549 15818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:31.647614 15818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:31.648381 15818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:31.648727 15818 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec1beef6b2d493d9183ce0ff0332260; no voters:
I20260619 16:38:31.649027 15818 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:38:31.649165 15823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:31.649398 15823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec1beef6b2d493d9183ce0ff0332260, State: Running, Role: LEADER
I20260619 16:38:31.649713 15823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:31.650180 15818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:31.651374 15825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec1beef6b2d493d9183ce0ff0332260. Latest consensus state: current_term: 1 leader_uuid: "7ec1beef6b2d493d9183ce0ff0332260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260619 16:38:31.651397 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec1beef6b2d493d9183ce0ff0332260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260619 16:38:31.651510 15825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:31.651510 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:31.651937 15832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:31.654590 15832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:31.659665 15832 catalog_manager.cc:1347] Generated new cluster ID: 6d8690df104c4981a2ba1dd41d35558c
I20260619 16:38:31.659723 15832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:31.668695 15832 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:38:31.669549 15832 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:31.685562 15832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Generated new TSK 0
I20260619 16:38:31.686179 15832 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:38:31.720504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36777
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:31.827837 15844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:31.828058 15844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:31.828096 15844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:31.832077 15844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:31.832121 15844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:31.832181 15844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:31.837141 15844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:31.838059 15844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:31.838937 15844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:38:31.849114 15844 server_base.cc:1034] running on GCE node
W20260619 16:38:31.849077 15849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:31.849040 15850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:31.849036 15852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:31.849763 15844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:31.850384 15844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:31.851577 15844 hybrid_clock.cc:648] HybridClock initialized: now 1781887111851530 us; error 62 us; skew 500 ppm
Jun 19 16:38:31 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:31.854152 15844 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:31.854983 15844 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260619 16:38:31.855525 15844 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:31.855611 15844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:31.855829 15844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:31.857829 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "b5024fe0fcb74bb7a093f787036a3120"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "8bf6b7eb686eac3bd112391c149a3ddf"
server_key_iv: "7cac85fdaba07b359552ab2b66e3084e"
server_key_version: "encryptionkey@0"
I20260619 16:38:31.858347 15844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b5024fe0fcb74bb7a093f787036a3120"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "8bf6b7eb686eac3bd112391c149a3ddf"
server_key_iv: "7cac85fdaba07b359552ab2b66e3084e"
server_key_version: "encryptionkey@0"
I20260619 16:38:31.862109 15844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 16:38:31.864331 15859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:31.865643 15844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:31.865768 15844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "b5024fe0fcb74bb7a093f787036a3120"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "8bf6b7eb686eac3bd112391c149a3ddf"
server_key_iv: "7cac85fdaba07b359552ab2b66e3084e"
server_key_version: "encryptionkey@0"
I20260619 16:38:31.865870 15844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:31.893927 15844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:31.896644 15844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:31.896839 15844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:31.897442 15844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:31.898233 15844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:31.898304 15844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:31.898370 15844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:31.898419 15844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:31.906939 15844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44443
I20260619 16:38:31.906980 15972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44443 every 8 connection(s)
I20260619 16:38:31.907754 15844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
Jun 19 16:38:31 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:31.916523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15844
I20260619 16:38:31.916646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:38:31.916920 392 external_mini_cluster.cc:1428] Setting key a1dc9dc142448611fb3813363eb017f5
I20260619 16:38:31.920589 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:31.909468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43153 (local address 127.0.98.62:36777)
0619 16:38:31.909613 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:31.909617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:31.910634 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 16:38:31.911483 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:31.911487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:31.911543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:31.911619 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:31.912688 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:31.913239 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:31.914362 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:31.914606 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:31.914714 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:31.917131 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:31.917134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:31.917137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:31.917169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:31.918707 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:31.919174 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:31.919179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:31.919181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:31.919259 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:31.919586 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:31.919591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:31.919593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:31.919761 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:31.919863 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:31.920314 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 16:38:31.920428 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:38:31.921377 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:31.909871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:43153)
0619 16:38:31.910495 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:31.910534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:31.911287 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:31.911770 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:31.911777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:31.912103 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:31.912546 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:31.912558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:31.913379 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:31.913385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:31.914234 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:31.914243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:31.914747 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:31.914751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:31.914912 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:31.915374 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:31.915388 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:31.916939 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:31.918817 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:31.918822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:31.918835 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:31.919073 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:31.919352 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:31.919356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:31.919358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:31.919493 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:31.919859 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:31.919866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:31.920096 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 16:38:31.920274 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 16:38:31.920570 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":110,"threads_started":1}
I20260619 16:38:31.922633 15973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36777
I20260619 16:38:31.922848 15973 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:31.923296 15973 heartbeater.cc:511] Master 127.0.98.62:36777 requested a full tablet report, sending...
I20260619 16:38:31.924708 15782 ts_manager.cc:194] Registered new tserver with Master: b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443)
I20260619 16:38:31.925824 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43153
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:38:31.944108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36777
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:32.050259 15978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:32.050482 15978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:32.050527 15978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:32.054548 15978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:32.054597 15978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:32.054653 15978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:32.059576 15978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:32.060429 15978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:32.061299 15978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:32.070266 15983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:32.070508 15978 server_base.cc:1034] running on GCE node
W20260619 16:38:32.070264 15986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:32.070289 15984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:32.071019 15978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:32.071595 15978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:32.072818 15978 hybrid_clock.cc:648] HybridClock initialized: now 1781887112072787 us; error 39 us; skew 500 ppm
Jun 19 16:38:32 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:32.075464 15978 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:32.076282 15978 webserver.cc:460] Webserver started at http://127.0.98.2:43707/ using document root <none> and password file <none>
I20260619 16:38:32.076819 15978 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:32.076908 15978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:32.077127 15978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:32.079149 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "96a0d12ec2e44005857107b48e7adc5f"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "32f3064db3ba92c2fe26b3c1e0a490e1"
server_key_iv: "242483b770f04d2947258cff3a52c9b7"
server_key_version: "encryptionkey@0"
I20260619 16:38:32.079702 15978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96a0d12ec2e44005857107b48e7adc5f"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "32f3064db3ba92c2fe26b3c1e0a490e1"
server_key_iv: "242483b770f04d2947258cff3a52c9b7"
server_key_version: "encryptionkey@0"
I20260619 16:38:32.083287 15978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:38:32.085685 15993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:32.086828 15978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:32.086968 15978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "96a0d12ec2e44005857107b48e7adc5f"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "32f3064db3ba92c2fe26b3c1e0a490e1"
server_key_iv: "242483b770f04d2947258cff3a52c9b7"
server_key_version: "encryptionkey@0"
I20260619 16:38:32.087073 15978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:32.099421 15978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:32.102214 15978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:32.102420 15978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:32.103046 15978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:32.104172 15978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:32.104223 15978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:32.104264 15978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:32.104285 15978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:32.113889 15978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38505
I20260619 16:38:32.113901 16106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38505 every 8 connection(s)
I20260619 16:38:32.114774 15978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:38:32.119474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 15978
I20260619 16:38:32.119611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:38:32.119913 392 external_mini_cluster.cc:1428] Setting key 18d92c679990b8e8d40c99ebca8ebacb
Jun 19 16:38:32 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:38:32.130162 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.116527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43795 (local address 127.0.98.62:36777)
0619 16:38:32.116703 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.116709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.117738 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 16:38:32.118633 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.118637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.118698 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.118807 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:32.120434 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.121133 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.123043 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.123318 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.123451 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.126379 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:32.126382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:32.126384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:32.126408 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:32.128009 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:32.128554 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:32.128564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:32.128566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:32.128640 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:32.129018 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:32.129022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:32.129023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:32.129219 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:32.129329 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.129859 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.129989 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:38:32.131026 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.116882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:43795)
0619 16:38:32.117596 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.117636 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.118431 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.119022 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.119030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.119590 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:32.120276 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.120302 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.121262 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.121265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.122858 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.122873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.123510 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.123515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.123678 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.124331 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:32.124347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:32.126187 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:32.128166 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:32.128172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:32.128187 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:32.128441 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:32.128734 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:32.128738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:32.128740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:32.128908 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:32.129368 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:32.129377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:32.129619 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.129801 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.130131 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":117,"threads_started":1}
I20260619 16:38:32.132283 16107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:38:32.132547 16107 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:32.133193 16107 heartbeater.cc:511] Master 127.0.98.62:36777 requested a full tablet report, sending...
I20260619 16:38:32.134426 15782 ts_manager.cc:194] Registered new tserver with Master: 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505)
I20260619 16:38:32.135259 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:38:32.148588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36777
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:32.255862 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:32.256124 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:32.256209 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:32.260149 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:32.260223 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:32.260339 16112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:32.265384 16112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:32.266337 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:32.267297 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:38:32.276959 16112 server_base.cc:1034] running on GCE node
W20260619 16:38:32.276890 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:32.276980 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:32.276880 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:32.277606 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:32.278184 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:32.279350 16112 hybrid_clock.cc:648] HybridClock initialized: now 1781887112279321 us; error 45 us; skew 500 ppm
Jun 19 16:38:32 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:32.281997 16112 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:32.282861 16112 webserver.cc:460] Webserver started at http://127.0.98.3:33927/ using document root <none> and password file <none>
I20260619 16:38:32.283394 16112 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:32.283483 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:32.283708 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:38:32.285684 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "d870886ec7ab453d8bc89eba502011c1"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "8e809a41cf8c0a99d9b54c7633e94e47"
server_key_iv: "ae206b7306b012ceff0b168775a3ab6a"
server_key_version: "encryptionkey@0"
I20260619 16:38:32.286204 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d870886ec7ab453d8bc89eba502011c1"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "8e809a41cf8c0a99d9b54c7633e94e47"
server_key_iv: "ae206b7306b012ceff0b168775a3ab6a"
server_key_version: "encryptionkey@0"
I20260619 16:38:32.289857 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:38:32.292203 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:32.293453 16112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:32.293587 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "d870886ec7ab453d8bc89eba502011c1"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "8e809a41cf8c0a99d9b54c7633e94e47"
server_key_iv: "ae206b7306b012ceff0b168775a3ab6a"
server_key_version: "encryptionkey@0"
I20260619 16:38:32.293692 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:32.320919 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:32.323766 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:32.323969 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:32.324548 16112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:32.325346 16112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:32.325416 16112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:32.325484 16112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:32.325536 16112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:32.334349 16112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41419
I20260619 16:38:32.334378 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41419 every 8 connection(s)
I20260619 16:38:32.335203 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 16:38:32 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:32.344875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16112
I20260619 16:38:32.345067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:38:32.345425 392 external_mini_cluster.cc:1428] Setting key a4aab06be5a620b3f39f665c19c3646d
I20260619 16:38:32.348066 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.336837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34859 (local address 127.0.98.62:36777)
0619 16:38:32.337000 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.337005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.337991 (+ 986us) server_negotiation.cc:407] Connection header received
0619 16:38:32.338767 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.338770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.338815 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.338886 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:32.339947 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.340476 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.341704 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.341939 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.342028 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.344364 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:32.344367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:32.344369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:32.344392 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:32.346070 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:32.346575 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:32.346581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:32.346582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:32.346667 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:32.347017 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:32.347021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:32.347023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:32.347202 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:32.347310 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.347779 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.347902 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:38:32.348791 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.337198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:34859)
0619 16:38:32.337857 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.337894 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.338602 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.339043 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.339049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.339378 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:32.339808 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.339820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.340610 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.340613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.341573 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.341582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.342072 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.342075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.342209 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.342768 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:32.342784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:32.344185 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:32.346195 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:32.346201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:32.346213 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:32.346470 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:32.346769 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:32.346772 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:32.346774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:32.346911 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:32.347307 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:32.347315 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:32.347553 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.347736 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.348042 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":170,"threads_started":1}
I20260619 16:38:32.349843 16241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36777
I20260619 16:38:32.350116 16241 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:32.350522 16241 heartbeater.cc:511] Master 127.0.98.62:36777 requested a full tablet report, sending...
I20260619 16:38:32.351567 15782 ts_manager.cc:194] Registered new tserver with Master: d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419)
I20260619 16:38:32.352223 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34859
I20260619 16:38:32.359089 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:38:32.367933 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.360877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.0.98.62:36777)
0619 16:38:32.361022 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.361026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.361131 (+ 105us) server_negotiation.cc:407] Connection header received
0619 16:38:32.361221 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.361223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.361257 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.361373 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:32.362003 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.362483 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.363267 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.363473 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.363547 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.364908 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:32.364911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:32.364913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:32.364933 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:32.366402 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:32.366769 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:32.366772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:32.366773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:32.366840 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:32.367079 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:32.367081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:32.367082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:32.367276 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:32.367380 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.367637 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.367726 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 16:38:32.370728 15782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:38:32.374281 15782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:38:32.388219 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.382912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.0.98.3:41419)
0619 16:38:32.383274 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.383287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.384211 (+ 924us) server_negotiation.cc:407] Connection header received
0619 16:38:32.384272 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.384275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.384392 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.384536 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:32.384742 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.385639 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387035 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.387556 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387676 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.387814 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.387982 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.388073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":96,"threads_started":1}
I20260619 16:38:32.388208 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.382745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41419 (local address 127.0.0.1:44630)
0619 16:38:32.384023 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.384035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.384105 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.384517 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.384520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.384536 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:32.384623 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.384629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.385785 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.385788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.386926 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.386932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.387677 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.387779 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.387813 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.387951 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.388073 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1218,"mutex_wait_us":25,"thread_start_us":74,"threads_started":1}
I20260619 16:38:32.388412 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.382285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.0.1:57026)
0619 16:38:32.383232 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.383246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.383336 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.383970 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.383974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.383990 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:32.384087 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.384094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.385408 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.385412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.386663 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.386672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.387899 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.388071 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.388138 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.388211 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.388282 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":126,"thread_start_us":92,"threads_started":1}
I20260619 16:38:32.388499 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.382247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.0.98.1:44443)
0619 16:38:32.382559 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.382575 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.382926 (+ 351us) server_negotiation.cc:407] Connection header received
0619 16:38:32.383215 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.383222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.383419 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.383544 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:32.383946 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.385160 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.386850 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.387602 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387708 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.387834 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.388272 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.388358 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
I20260619 16:38:32.388852 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.382090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:38158)
0619 16:38:32.382797 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.382846 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.383040 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.383724 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.383728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.383745 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:32.383831 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.383835 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.385296 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.385299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.386753 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.386763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.387899 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.388066 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.388138 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.388663 (+ 525us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.388735 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":103,"thread_start_us":61,"threads_started":1}
I20260619 16:38:32.389451 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.382858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.0.98.2:38505)
0619 16:38:32.383599 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.383610 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.383632 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:38:32.383701 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.383704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.383825 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.383978 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:32.384177 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.385242 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.386787 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.387602 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.387708 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.387834 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.388661 (+ 827us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.388745 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":113,"threads_started":1}
I20260619 16:38:32.391753 16175 tablet_service.cc:1467] Processing CreateTablet for tablet bcf19bb26cc04e36abc153c0c6460fca (DEFAULT_TABLE table=test-table [id=ca993adf031e43a693553c6f47a3dcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:32.391754 15907 tablet_service.cc:1467] Processing CreateTablet for tablet bcf19bb26cc04e36abc153c0c6460fca (DEFAULT_TABLE table=test-table [id=ca993adf031e43a693553c6f47a3dcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:32.392853 16175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf19bb26cc04e36abc153c0c6460fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:32.392853 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf19bb26cc04e36abc153c0c6460fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:32.398658 16041 tablet_service.cc:1467] Processing CreateTablet for tablet bcf19bb26cc04e36abc153c0c6460fca (DEFAULT_TABLE table=test-table [id=ca993adf031e43a693553c6f47a3dcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:32.399502 16041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf19bb26cc04e36abc153c0c6460fca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:32.399937 16259 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Bootstrap starting.
I20260619 16:38:32.400449 16258 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Bootstrap starting.
I20260619 16:38:32.402448 16258 tablet_bootstrap.cc:654] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:32.402532 16259 tablet_bootstrap.cc:654] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:32.403546 16258 log.cc:824] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:32.403767 16259 log.cc:824] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:32.405220 16258 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: No bootstrap required, opened a new log
I20260619 16:38:32.405436 16258 ts_tablet_manager.cc:1397] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:38:32.405680 16259 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: No bootstrap required, opened a new log
I20260619 16:38:32.405862 16259 ts_tablet_manager.cc:1397] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 16:38:32.407512 16262 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Bootstrap starting.
I20260619 16:38:32.408741 16258 raft_consensus.cc:348] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.408973 16258 raft_consensus.cc:374] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:32.409044 16258 raft_consensus.cc:729] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5024fe0fcb74bb7a093f787036a3120, State: Initialized, Role: FOLLOWER
I20260619 16:38:32.409516 16258 consensus_queue.cc:260] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.409484 16259 raft_consensus.cc:348] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.409818 16259 raft_consensus.cc:374] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:32.409899 16259 raft_consensus.cc:729] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870886ec7ab453d8bc89eba502011c1, State: Initialized, Role: FOLLOWER
I20260619 16:38:32.410037 16262 tablet_bootstrap.cc:654] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:32.410393 16258 ts_tablet_manager.cc:1428] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:38:32.410373 16259 consensus_queue.cc:260] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.410750 15973 heartbeater.cc:503] Master 127.0.98.62:36777 was elected leader, sending a full tablet report...
I20260619 16:38:32.410848 16262 log.cc:824] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:32.411324 16259 ts_tablet_manager.cc:1428] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:38:32.411527 16241 heartbeater.cc:503] Master 127.0.98.62:36777 was elected leader, sending a full tablet report...
I20260619 16:38:32.413545 16262 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: No bootstrap required, opened a new log
I20260619 16:38:32.413775 16262 ts_tablet_manager.cc:1397] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 16:38:32.416785 16262 raft_consensus.cc:348] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.416986 16262 raft_consensus.cc:374] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:32.417019 16262 raft_consensus.cc:729] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a0d12ec2e44005857107b48e7adc5f, State: Initialized, Role: FOLLOWER
I20260619 16:38:32.417368 16262 consensus_queue.cc:260] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.417914 16107 heartbeater.cc:503] Master 127.0.98.62:36777 was elected leader, sending a full tablet report...
I20260619 16:38:32.418111 16262 ts_tablet_manager.cc:1428] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 16:38:32.586156 16242 tablet.cc:2369] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:38:32.615813 16108 tablet.cc:2369] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:38:32.659020 15974 tablet.cc:2369] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:32.669292 16264 raft_consensus.cc:482] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:32.669574 16264 raft_consensus.cc:504] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.670841 16264 leader_election.cc:288] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443), 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505)
I20260619 16:38:32.677302 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.671119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.98.3:57343)
0619 16:38:32.671238 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.671253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.671361 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.671634 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.671637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.671654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:32.671777 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.671782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.672872 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.672875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.675868 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.675902 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.676782 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.676786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.676912 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.676944 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.677049 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.677140 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 16:38:32.677384 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.671125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.98.3:46963)
0619 16:38:32.671740 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.671752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:32.671832 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:32.672095 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:32.672098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:32.672116 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:32.672226 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.672232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.673212 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.673215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.675927 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:32.675945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.676895 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.676898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:32.677031 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.677057 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 16:38:32.677157 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 16:38:32.677224 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":1781,"thread_start_us":143,"threads_started":1}
I20260619 16:38:32.677394 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.671172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46963 (local address 127.0.98.1:44443)
0619 16:38:32.671293 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.671297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.671834 (+ 537us) server_negotiation.cc:407] Connection header received
0619 16:38:32.671955 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.671958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.672000 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.672057 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:32.672347 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.673079 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.676078 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.676660 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.676736 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.676803 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.677192 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.677270 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:38:32.678122 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.671183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57343 (local address 127.0.98.2:38505)
0619 16:38:32.671317 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.671320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.671334 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:38:32.671481 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.671483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.671528 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.671625 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:32.671936 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.672721 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.676025 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.676646 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.676754 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.676805 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.677889 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.677993 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:38:32.678280 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf19bb26cc04e36abc153c0c6460fca" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5024fe0fcb74bb7a093f787036a3120" is_pre_election: true
I20260619 16:38:32.678647 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf19bb26cc04e36abc153c0c6460fca" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a0d12ec2e44005857107b48e7adc5f" is_pre_election: true
I20260619 16:38:32.678650 15927 raft_consensus.cc:2454] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 0.
I20260619 16:38:32.678918 16061 raft_consensus.cc:2454] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 0.
I20260619 16:38:32.679339 16130 leader_election.cc:302] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 96a0d12ec2e44005857107b48e7adc5f, b5024fe0fcb74bb7a093f787036a3120, d870886ec7ab453d8bc89eba502011c1; no voters:
I20260619 16:38:32.679622 16264 raft_consensus.cc:2781] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:38:32.679723 16264 raft_consensus.cc:482] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:38:32.679780 16264 raft_consensus.cc:3037] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:32.681171 16264 raft_consensus.cc:504] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.681561 16264 leader_election.cc:288] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 election: Requested vote from peers b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443), 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505)
I20260619 16:38:32.681973 16061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf19bb26cc04e36abc153c0c6460fca" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a0d12ec2e44005857107b48e7adc5f"
I20260619 16:38:32.682031 15927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf19bb26cc04e36abc153c0c6460fca" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5024fe0fcb74bb7a093f787036a3120"
I20260619 16:38:32.682152 15927 raft_consensus.cc:3037] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:32.682144 16061 raft_consensus.cc:3037] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:32.683378 15927 raft_consensus.cc:2454] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 1.
I20260619 16:38:32.683377 16061 raft_consensus.cc:2454] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 1.
I20260619 16:38:32.683740 16129 leader_election.cc:302] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1; no voters:
I20260619 16:38:32.683946 16264 raft_consensus.cc:2781] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:32.684156 16264 raft_consensus.cc:686] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 1 LEADER]: Becoming Leader. State: Replica: d870886ec7ab453d8bc89eba502011c1, State: Running, Role: LEADER
I20260619 16:38:32.684484 16264 consensus_queue.cc:237] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:32.687824 15782 catalog_manager.cc:5495] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870886ec7ab453d8bc89eba502011c1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d870886ec7ab453d8bc89eba502011c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } health_report { overall_health: HEALTHY } } }
I20260619 16:38:32.752547 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.748801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.0.98.3:41419)
0619 16:38:32.748944 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:32.748949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:32.749008 (+ 59us) server_negotiation.cc:407] Connection header received
0619 16:38:32.749097 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:32.749100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:32.749156 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:32.749265 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:32.749543 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.750130 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.751100 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:32.751332 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:32.751424 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:32.751711 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:32.752213 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:38:32.752319 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:32.752372 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 16:38:32.752426 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:38:32.758025 15927 raft_consensus.cc:1261] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Refusing update from remote peer d870886ec7ab453d8bc89eba502011c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:32.758028 16061 raft_consensus.cc:1261] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Refusing update from remote peer d870886ec7ab453d8bc89eba502011c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:32.758715 16264 consensus_queue.cc:1035] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:32.758944 16268 consensus_queue.cc:1035] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:32.765422 16271 mvcc.cc:205] Tried to move back new op lower bound from 7298609613849763840 to 7298609613558267904. Current Snapshot: MvccSnapshot[applied={T|T < 7298609613849763840}]
I20260619 16:38:32.767503 16272 mvcc.cc:205] Tried to move back new op lower bound from 7298609613849763840 to 7298609613558267904. Current Snapshot: MvccSnapshot[applied={T|T < 7298609613849763840}]
I20260619 16:38:32.773819 15782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:32.773983 15782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:32.776315 15782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Sending DeleteTablet for 3 replicas of tablet bcf19bb26cc04e36abc153c0c6460fca
I20260619 16:38:32.777026 16041 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf19bb26cc04e36abc153c0c6460fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57026
I20260619 16:38:32.777119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15844
I20260619 16:38:32.777060 15907 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf19bb26cc04e36abc153c0c6460fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38158
I20260619 16:38:32.778327 16281 tablet_replica.cc:330] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: stopping tablet replica
I20260619 16:38:32.778285 16175 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf19bb26cc04e36abc153c0c6460fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44630
I20260619 16:38:32.778620 16281 raft_consensus.cc:2229] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:32.778908 16281 raft_consensus.cc:2258] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:32.780359 16281 ts_tablet_manager.cc:1905] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:32.781522 16282 tablet_replica.cc:330] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: stopping tablet replica
I20260619 16:38:32.781890 16282 raft_consensus.cc:2229] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:38:32.782284 16282 raft_consensus.cc:2258] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:32.783222 16281 ts_tablet_manager.cc:1918] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:32.783368 16281 log.cc:1196] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/bcf19bb26cc04e36abc153c0c6460fca
I20260619 16:38:32.783605 16282 ts_tablet_manager.cc:1905] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 16:38:32.784003 15768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44443 (error 108)
I20260619 16:38:32.784052 16281 ts_tablet_manager.cc:1939] T bcf19bb26cc04e36abc153c0c6460fca P 96a0d12ec2e44005857107b48e7adc5f: Deleting consensus metadata
I20260619 16:38:32.785142 15767 catalog_manager.cc:4841] TS 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505): tablet bcf19bb26cc04e36abc153c0c6460fca (table test-table [id=ca993adf031e43a693553c6f47a3dcc2]) successfully deleted
I20260619 16:38:32.785506 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:32.785145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:38168)
0619 16:38:32.785351 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:32.785420 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 16:38:32.785758 15768 catalog_manager.cc:4584] TS b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcf19bb26cc04e36abc153c0c6460fca: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
I20260619 16:38:32.786058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15978
I20260619 16:38:32.787043 16282 ts_tablet_manager.cc:1918] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:32.787155 16282 log.cc:1196] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/bcf19bb26cc04e36abc153c0c6460fca
I20260619 16:38:32.787514 16282 ts_tablet_manager.cc:1939] T bcf19bb26cc04e36abc153c0c6460fca P d870886ec7ab453d8bc89eba502011c1: Deleting consensus metadata
I20260619 16:38:32.788602 15768 catalog_manager.cc:4841] TS d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419): tablet bcf19bb26cc04e36abc153c0c6460fca (table test-table [id=ca993adf031e43a693553c6f47a3dcc2]) successfully deleted
I20260619 16:38:32.792601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 16112
I20260619 16:38:32.798590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 15750
I20260619 16:38:52.805017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--webserver_interface=127.0.98.62
--webserver_port=42541
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:38:52.914835 16284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:52.915074 16284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:52.915112 16284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:52.919099 16284 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:38:52.919143 16284 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:38:52.919158 16284 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:38:52.919173 16284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:38:52.919225 16284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:52.919261 16284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:52.919289 16284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:38:52.919322 16284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:38:52.925783 16284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:52.926750 16284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:52.927735 16284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:52.940295 16290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:52.940297 16293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:52.940306 16291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:52.940654 16284 server_base.cc:1034] running on GCE node
I20260619 16:38:52.941205 16284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:52.942386 16284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:52.943557 16284 hybrid_clock.cc:648] HybridClock initialized: now 1781887132943536 us; error 27 us; skew 500 ppm
Jun 19 16:38:52 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:52.946373 16284 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:38:52.947232 16284 webserver.cc:460] Webserver started at http://127.0.98.62:42541/ using document root <none> and password file <none>
I20260619 16:38:52.947747 16284 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:52.947814 16284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:52.951545 16284 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:52.953732 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:52.954982 16284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:52.955124 16284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "7ec1beef6b2d493d9183ce0ff0332260"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "d0937e9d610344bc3ac30647ab03a767"
server_key_iv: "6f5c23cb210050d8c853aa748558c113"
server_key_version: "encryptionkey@0"
I20260619 16:38:52.955700 16284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:52.975968 16284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:52.979338 16284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:52.979552 16284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:52.986946 16284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36777
I20260619 16:38:52.987267 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36777 every 8 connection(s)
I20260619 16:38:52.988030 16284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:38:52.991119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16284
I20260619 16:38:52.991554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36777
--webserver_port=41187
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:52.992470 16353 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:38:52.994115 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap starting.
I20260619 16:38:53.010533 16353 log.cc:824] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:53.018852 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:53.019214 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap complete.
I20260619 16:38:53.023258 16353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:53.023664 16353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec1beef6b2d493d9183ce0ff0332260, State: Initialized, Role: FOLLOWER
I20260619 16:38:53.024186 16353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:53.024343 16353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:38:53.024435 16353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:38:53.024550 16353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 1 FOLLOWER]: Advancing to term 2
I20260619 16:38:53.026511 16353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:53.026839 16353 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec1beef6b2d493d9183ce0ff0332260; no voters:
I20260619 16:38:53.027136 16353 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 16:38:53.027281 16357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 16:38:53.027527 16357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 LEADER]: Becoming Leader. State: Replica: 7ec1beef6b2d493d9183ce0ff0332260, State: Running, Role: LEADER
I20260619 16:38:53.027894 16357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:38:53.028317 16353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:38:53.029357 16358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7ec1beef6b2d493d9183ce0ff0332260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260619 16:38:53.029485 16358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:53.029831 16366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:38:53.029826 16359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec1beef6b2d493d9183ce0ff0332260. Latest consensus state: current_term: 2 leader_uuid: "7ec1beef6b2d493d9183ce0ff0332260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260619 16:38:53.029991 16359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: This master's current role is: LEADER
I20260619 16:38:53.034633 16366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:38:53.035400 16366 catalog_manager.cc:1259] Loaded cluster ID: 6d8690df104c4981a2ba1dd41d35558c
I20260619 16:38:53.035574 16366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:38:53.038203 16366 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:38:53.038874 16366 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Loaded TSK: 0
I20260619 16:38:53.058413 16366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Generated new TSK 1
I20260619 16:38:53.058681 16366 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:38:53.109719 16355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:53.109948 16355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:53.109989 16355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:53.113927 16355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:53.113974 16355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:53.114054 16355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:38:53.119159 16355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41187
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:53.120055 16355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:53.120980 16355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:53.131611 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:53.131647 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:53.131982 16355 server_base.cc:1034] running on GCE node
W20260619 16:38:53.131989 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:53.132480 16355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:53.133144 16355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:53.134326 16355 hybrid_clock.cc:648] HybridClock initialized: now 1781887133134312 us; error 69 us; skew 500 ppm
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:53.137215 16355 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:38:53.138074 16355 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260619 16:38:53.138633 16355 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:53.138736 16355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:53.142608 16355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:38:53.144809 16390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.146078 16355 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:53.146216 16355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "b5024fe0fcb74bb7a093f787036a3120"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "8bf6b7eb686eac3bd112391c149a3ddf"
server_key_iv: "7cac85fdaba07b359552ab2b66e3084e"
server_key_version: "encryptionkey@0"
I20260619 16:38:53.146757 16355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:53.158180 16355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:53.161543 16355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:53.161733 16355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:53.162285 16355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:53.163367 16397 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:38:53.168323 16355 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:38:53.168500 16355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 16:38:53.168586 16355 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:38:53.170267 16355 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:38:53.170334 16355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 16:38:53.170588 16397 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Bootstrap starting.
I20260619 16:38:53.181234 16355 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44443
I20260619 16:38:53.181273 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44443 every 8 connection(s)
I20260619 16:38:53.182224 16355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:38:53.188050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16355
I20260619 16:38:53.188560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38505
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36777
--webserver_port=43707
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:53.190512 16397 log.cc:824] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:53.198524 16397 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:38:53.198953 16397 tablet_bootstrap.cc:492] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Bootstrap complete.
I20260619 16:38:53.200224 16397 ts_tablet_manager.cc:1397] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.013s
I20260619 16:38:53.201287 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.184437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39789 (local address 127.0.98.62:36777)
0619 16:38:53.184822 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.184841 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.185134 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:38:53.186281 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.186297 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.186680 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.187126 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:53.188326 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.189975 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.191400 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.192162 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.192363 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.195451 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:53.195458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:53.195474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:53.195505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:53.197653 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.198326 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.198331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.198336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.198483 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.198915 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.198919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.198922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.199376 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:53.199511 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.200192 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.200484 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":113,"threads_started":1}
I20260619 16:38:53.201351 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.184318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:39789)
0619 16:38:53.184956 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.185006 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.185853 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.187150 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.187156 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.187513 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:53.188115 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.188138 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.190166 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.190168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.191255 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.191267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.192317 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.192321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.192496 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.193206 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:53.193225 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:53.195221 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:53.197781 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.197791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.197805 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.198181 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.198595 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.198600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.198605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.198796 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.199507 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:53.199515 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:53.199838 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.200065 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.200421 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":152,"threads_started":1}
I20260619 16:38:53.203032 16397 raft_consensus.cc:348] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:53.203580 16397 raft_consensus.cc:729] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5024fe0fcb74bb7a093f787036a3120, State: Initialized, Role: FOLLOWER
I20260619 16:38:53.204059 16397 consensus_queue.cc:260] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } } peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } }
I20260619 16:38:53.204514 16505 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36777
I20260619 16:38:53.204717 16505 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:53.205085 16397 ts_tablet_manager.cc:1428] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:38:53.205219 16505 heartbeater.cc:511] Master 127.0.98.62:36777 requested a full tablet report, sending...
I20260619 16:38:53.207417 16317 ts_manager.cc:194] Registered new tserver with Master: b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443)
I20260619 16:38:53.210628 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39789
I20260619 16:38:53.212896 16505 heartbeater.cc:503] Master 127.0.98.62:36777 was elected leader, sending a full tablet report...
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 16:38:53.220515 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.210091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:43250)
0619 16:38:53.210598 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.210640 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.210768 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.211249 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.211254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.211652 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:53.211775 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.211783 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.213193 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.213196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.214145 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.214156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.214862 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.214875 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.214960 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.215492 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:53.215504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:53.216947 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:53.218961 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.218969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.218972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.219188 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.219498 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.219501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.219502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.219584 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.220048 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:53.220053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:53.220146 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.220257 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.220386 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":91,"threads_started":1}
I20260619 16:38:53.220850 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.210167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.98.1:44443)
0619 16:38:53.210779 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.210793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.210816 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:38:53.210901 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.210905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.211098 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.211226 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:53.211904 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.213073 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.214281 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.214758 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.214863 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.217081 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:53.217091 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:53.217102 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:53.217126 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:53.218841 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.219294 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.219300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.219302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.219391 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.219670 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.219673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.219675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.219930 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:53.220052 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.220352 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.220508 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":526,"thread_start_us":82,"threads_started":1}
I20260619 16:38:53.221732 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf19bb26cc04e36abc153c0c6460fca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43250
I20260619 16:38:53.222565 16514 tablet_replica.cc:330] stopping tablet replica
I20260619 16:38:53.222844 16514 raft_consensus.cc:2229] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:53.223067 16514 raft_consensus.cc:2258] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:53.224040 16514 ts_tablet_manager.cc:1905] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:53.228320 16514 ts_tablet_manager.cc:1918] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:53.228427 16514 log.cc:1196] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal/wals/bcf19bb26cc04e36abc153c0c6460fca
I20260619 16:38:53.228757 16514 ts_tablet_manager.cc:1939] T bcf19bb26cc04e36abc153c0c6460fca P b5024fe0fcb74bb7a093f787036a3120: Deleting consensus metadata
I20260619 16:38:53.230644 16303 catalog_manager.cc:4841] TS b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443): tablet bcf19bb26cc04e36abc153c0c6460fca (table test-table [id=ca993adf031e43a693553c6f47a3dcc2]) successfully deleted
W20260619 16:38:53.304019 16509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:53.304293 16509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:53.304365 16509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:53.308316 16509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:53.308408 16509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:53.308501 16509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:38:53.313627 16509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43707
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:53.314651 16509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:53.315629 16509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:53.324937 16520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:53.324959 16522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:53.324898 16519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:53.325127 16509 server_base.cc:1034] running on GCE node
I20260619 16:38:53.325547 16509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:53.326144 16509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:53.327303 16509 hybrid_clock.cc:648] HybridClock initialized: now 1781887133327283 us; error 30 us; skew 500 ppm
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:53.329936 16509 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:38:53.330765 16509 webserver.cc:460] Webserver started at http://127.0.98.2:43707/ using document root <none> and password file <none>
I20260619 16:38:53.331248 16509 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:53.331312 16509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:53.334789 16509 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:53.336772 16529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.338080 16509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:53.338212 16509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "96a0d12ec2e44005857107b48e7adc5f"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "32f3064db3ba92c2fe26b3c1e0a490e1"
server_key_iv: "242483b770f04d2947258cff3a52c9b7"
server_key_version: "encryptionkey@0"
I20260619 16:38:53.338773 16509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:53.361433 16509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:53.364171 16509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:53.364359 16509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:53.364904 16509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:53.365748 16509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:53.365815 16509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.365877 16509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:53.365931 16509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.375016 16509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38505
I20260619 16:38:53.375041 16642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38505 every 8 connection(s)
I20260619 16:38:53.375921 16509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:53.385576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16509
I20260619 16:38:53.386008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41419
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36777
--webserver_port=33927
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:38:53.388795 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.377710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38159 (local address 127.0.98.62:36777)
0619 16:38:53.377856 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.377861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.378888 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 16:38:53.379673 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.379675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.379718 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.379790 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:53.380929 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.381469 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.382606 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.382825 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.382911 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.385212 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:53.385215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:53.385217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:53.385242 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:53.386753 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.387401 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.387405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.387409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.387503 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.387814 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.387819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.387821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.388059 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:53.388159 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.388577 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.388677 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:38:53.389611 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.378114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:38159)
0619 16:38:53.378754 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.378792 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.379505 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.379943 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.379949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.380279 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:53.380793 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.380808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.381588 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.381591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.382485 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.382494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.382948 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.382951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.383083 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.383651 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:53.383666 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:53.385064 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:53.386863 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.386871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.386888 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.387278 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.387597 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.387599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.387601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.387725 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.388154 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:53.388158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:53.388358 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.388504 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.388725 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":108,"threads_started":1}
I20260619 16:38:53.390764 16643 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36777
I20260619 16:38:53.391036 16643 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:53.391618 16643 heartbeater.cc:511] Master 127.0.98.62:36777 requested a full tablet report, sending...
I20260619 16:38:53.392769 16317 ts_manager.cc:194] Registered new tserver with Master: 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505)
I20260619 16:38:53.393362 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38159
W20260619 16:38:53.494640 16646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:38:53.494859 16646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:38:53.494899 16646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:38:53.498785 16646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:38:53.498831 16646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:38:53.498927 16646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:38:53.503963 16646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33927
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:38:53.504902 16646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:38:53.505861 16646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:38:53.515384 16654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:53.515401 16651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:38:53.515384 16652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:38:53.515488 16646 server_base.cc:1034] running on GCE node
I20260619 16:38:53.515959 16646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:38:53.516490 16646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:38:53.517678 16646 hybrid_clock.cc:648] HybridClock initialized: now 1781887133517647 us; error 43 us; skew 500 ppm
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:38:53.520246 16646 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:38:53.521080 16646 webserver.cc:460] Webserver started at http://127.0.98.3:33927/ using document root <none> and password file <none>
I20260619 16:38:53.521664 16646 fs_manager.cc:359] Metadata directory not provided
I20260619 16:38:53.521766 16646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:38:53.525312 16646 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:38:53.527382 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.528553 16646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:38:53.528677 16646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "d870886ec7ab453d8bc89eba502011c1"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "8e809a41cf8c0a99d9b54c7633e94e47"
server_key_iv: "ae206b7306b012ceff0b168775a3ab6a"
server_key_version: "encryptionkey@0"
I20260619 16:38:53.529186 16646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:38:53.552354 16646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:38:53.555433 16646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:38:53.555619 16646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:38:53.556139 16646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:38:53.556896 16646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:38:53.556965 16646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.557029 16646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:38:53.557078 16646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:38:53.565968 16646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41419
I20260619 16:38:53.566030 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41419 every 8 connection(s)
I20260619 16:38:53.566938 16646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:38:53.572779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16646
Jun 19 16:38:53 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:38:53.579856 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.568602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42883 (local address 127.0.98.62:36777)
0619 16:38:53.568757 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.568761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.569809 (+ 1048us) server_negotiation.cc:407] Connection header received
0619 16:38:53.570678 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.570681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.570747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.570835 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:53.571952 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.572468 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.573886 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.574116 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.574227 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.576468 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:53.576477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:53.576479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:53.576502 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:53.577955 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.578465 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.578469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.578471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.578548 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.578864 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.578868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.578870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.579037 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:53.579140 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.579587 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.579694 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:38:53.580780 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.568975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:42883)
0619 16:38:53.569670 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.569709 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.570475 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.570998 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.571005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.571329 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:38:53.571768 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.571780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.572590 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.572596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.573766 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.573775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.574258 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.574262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.574425 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.574974 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:38:53.574989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:38:53.576308 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:38:53.578069 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.578075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.578087 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.578326 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.578638 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:38:53.578642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:38:53.578644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:38:53.578773 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:38:53.579139 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:38:53.579146 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:38:53.579366 (+ 220us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.579546 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.579831 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":103,"threads_started":1}
I20260619 16:38:53.580953 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.573444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.98.62:36777)
0619 16:38:53.573644 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.573649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.573812 (+ 163us) server_negotiation.cc:407] Connection header received
0619 16:38:53.574001 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.574006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.574051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.574143 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:53.574825 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.575320 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.576226 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.576457 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.576563 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.577923 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:53.577929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:53.577936 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:53.577960 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:53.579389 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.579794 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.579799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.579801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.579881 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.580192 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.580196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.580198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.580384 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:53.580578 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.580677 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.580788 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260619 16:38:53.581904 16775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36777
I20260619 16:38:53.582139 16775 heartbeater.cc:464] Registering TS with master...
I20260619 16:38:53.582598 16775 heartbeater.cc:511] Master 127.0.98.62:36777 requested a full tablet report, sending...
I20260619 16:38:53.583527 16317 ts_manager.cc:194] Registered new tserver with Master: d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419)
I20260619 16:38:53.584074 16317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42883
I20260619 16:38:53.594604 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:38:53.598591 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.595231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.0.98.62:36777)
0619 16:38:53.595352 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.595356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.595580 (+ 224us) server_negotiation.cc:407] Connection header received
0619 16:38:53.595679 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.595681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.595767 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.595853 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:53.596199 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.596696 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.597653 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.597869 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.597954 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.598196 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:53.598313 (+ 117us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 16:38:53.598507 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45464: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:38:53.598665 16778 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45464: token expired
W20260619 16:38:53.598976 16245 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36777)
I20260619 16:38:53.608573 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.599313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.0.98.62:36777)
0619 16:38:53.599417 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.599423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.599438 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:38:53.599499 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.599501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.599531 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.599593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:38:53.600157 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.600601 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.601462 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.601655 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.601732 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.605959 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:38:53.605962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:38:53.605964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:38:53.605983 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:38:53.607291 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.607646 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.607649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.607650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.607718 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:38:53.607951 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:38:53.607953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:38:53.607954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:38:53.608098 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:38:53.608189 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.608393 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.608473 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:38:53.610694 16317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:38:53.613854 16317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:38:53.621397 16439 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7c8b338f1c41229c80bc3022605b7f (DEFAULT_TABLE table=test-table [id=508c9fc507bf4421938e35e4e274ddf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:53.622164 16439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7c8b338f1c41229c80bc3022605b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:53.625083 16397 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Bootstrap starting.
I20260619 16:38:53.625193 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.620196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.0.1:44066)
0619 16:38:53.620297 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.620312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.620383 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.621349 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.621355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.621386 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:53.621514 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.621520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.622916 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.622919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.623840 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.623848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.624788 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.624792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.624902 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.624946 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.624997 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.625065 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":6}
I20260619 16:38:53.625892 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.620395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.0.98.2:38505)
0619 16:38:53.620827 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.620840 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.620863 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:38:53.620937 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.620942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.621150 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.621347 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:53.621658 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.622792 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.623963 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.624540 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.625450 (+ 910us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.625592 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.625647 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.625741 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":130,"threads_started":1}
I20260619 16:38:53.625944 16397 tablet_bootstrap.cc:654] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:53.626174 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.620433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41419 (local address 127.0.0.1:47812)
0619 16:38:53.620680 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.620693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.620773 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.621434 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.621437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.621449 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:53.621687 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.621693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.623291 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.623294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.624388 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.624411 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.625737 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.625741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.625844 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.625884 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.625964 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.626035 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":150,"thread_start_us":104,"threads_started":1}
I20260619 16:38:53.626628 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.620739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.0.98.3:41419)
0619 16:38:53.621026 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.621035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.621054 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:38:53.621111 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.621114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.621242 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.621518 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:53.621794 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.623170 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.624536 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.625615 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.625717 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.625833 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.626369 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.626474 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
I20260619 16:38:53.627597 16397 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: No bootstrap required, opened a new log
I20260619 16:38:53.627684 16397 ts_tablet_manager.cc:1397] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 16:38:53.628099 16397 raft_consensus.cc:348] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.628244 16397 raft_consensus.cc:374] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:53.628290 16397 raft_consensus.cc:729] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5024fe0fcb74bb7a093f787036a3120, State: Initialized, Role: FOLLOWER
I20260619 16:38:53.628409 16397 consensus_queue.cc:260] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.628748 16397 ts_tablet_manager.cc:1428] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 16:38:53.629745 16577 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7c8b338f1c41229c80bc3022605b7f (DEFAULT_TABLE table=test-table [id=508c9fc507bf4421938e35e4e274ddf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:53.630721 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7c8b338f1c41229c80bc3022605b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:53.630656 16709 tablet_service.cc:1467] Processing CreateTablet for tablet 5c7c8b338f1c41229c80bc3022605b7f (DEFAULT_TABLE table=test-table [id=508c9fc507bf4421938e35e4e274ddf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:38:53.631503 16709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c7c8b338f1c41229c80bc3022605b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:38:53.637470 16788 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Bootstrap starting.
I20260619 16:38:53.638418 16789 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Bootstrap starting.
I20260619 16:38:53.639660 16788 tablet_bootstrap.cc:654] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:53.640964 16788 log.cc:824] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:53.641064 16789 tablet_bootstrap.cc:654] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Neither blocks nor log segments found. Creating new log.
I20260619 16:38:53.641913 16789 log.cc:824] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Log is configured to *not* fsync() on all Append() calls
I20260619 16:38:53.643234 16788 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: No bootstrap required, opened a new log
I20260619 16:38:53.643419 16788 ts_tablet_manager.cc:1397] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:38:53.644023 16789 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: No bootstrap required, opened a new log
I20260619 16:38:53.644217 16789 ts_tablet_manager.cc:1397] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 16:38:53.647221 16788 raft_consensus.cc:348] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.647492 16788 raft_consensus.cc:374] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:53.647543 16788 raft_consensus.cc:729] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d870886ec7ab453d8bc89eba502011c1, State: Initialized, Role: FOLLOWER
I20260619 16:38:53.647995 16788 consensus_queue.cc:260] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.647938 16789 raft_consensus.cc:348] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.648166 16789 raft_consensus.cc:374] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:38:53.648228 16789 raft_consensus.cc:729] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a0d12ec2e44005857107b48e7adc5f, State: Initialized, Role: FOLLOWER
I20260619 16:38:53.648617 16775 heartbeater.cc:503] Master 127.0.98.62:36777 was elected leader, sending a full tablet report...
I20260619 16:38:53.648700 16789 consensus_queue.cc:260] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.649056 16788 ts_tablet_manager.cc:1428] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:38:53.649636 16643 heartbeater.cc:503] Master 127.0.98.62:36777 was elected leader, sending a full tablet report...
I20260619 16:38:53.649659 16789 ts_tablet_manager.cc:1428] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260619 16:38:53.683408 16506 tablet.cc:2369] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:38:53.730319 16792 raft_consensus.cc:482] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:38:53.730527 16792 raft_consensus.cc:504] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.731737 16792 leader_election.cc:288] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505), b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443)
I20260619 16:38:53.735482 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.731941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.98.3:41859)
0619 16:38:53.732091 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.732107 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.732198 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.732507 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.732512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.732534 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:53.732658 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.732669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.733524 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.733530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.734404 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.734413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.735055 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.735058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.735160 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.735182 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.735277 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.735350 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:38:53.735482 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.732047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41859 (local address 127.0.98.2:38505)
0619 16:38:53.732180 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.732184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.732198 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:38:53.732330 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.732333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.732389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.732480 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:53.732786 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.733435 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.734524 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.734941 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.735056 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.735121 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.735291 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.735357 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:38:53.735986 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.731987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.98.3:46205)
0619 16:38:53.732679 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.732703 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 16:38:53.732814 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:38:53.733114 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:38:53.733117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:38:53.733133 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:38:53.733267 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.733278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.734023 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.734029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.734796 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:38:53.734804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.735513 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.735517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:38:53.735708 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.735729 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:38:53.735790 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 16:38:53.735850 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":345,"thread_start_us":117,"threads_started":1}
I20260619 16:38:53.736156 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7c8b338f1c41229c80bc3022605b7f" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a0d12ec2e44005857107b48e7adc5f" is_pre_election: true
I20260619 16:38:53.736371 16794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.732181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46205 (local address 127.0.98.1:44443)
0619 16:38:53.732464 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.732469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.732782 (+ 313us) server_negotiation.cc:407] Connection header received
0619 16:38:53.732926 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.732929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.732994 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.733105 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:38:53.733419 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.733913 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.734925 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.735372 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.735507 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.736103 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.736147 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.736228 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260619 16:38:53.736476 16597 raft_consensus.cc:2454] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 0.
I20260619 16:38:53.736867 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7c8b338f1c41229c80bc3022605b7f" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5024fe0fcb74bb7a093f787036a3120" is_pre_election: true
I20260619 16:38:53.736927 16663 leader_election.cc:302] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1; no voters:
I20260619 16:38:53.737147 16459 raft_consensus.cc:2454] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 0.
I20260619 16:38:53.737192 16792 raft_consensus.cc:2781] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:38:53.737344 16792 raft_consensus.cc:482] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:38:53.737412 16792 raft_consensus.cc:3037] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:53.738490 16792 raft_consensus.cc:504] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.738883 16792 leader_election.cc:288] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 election: Requested vote from peers 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505), b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443)
I20260619 16:38:53.739259 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7c8b338f1c41229c80bc3022605b7f" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5024fe0fcb74bb7a093f787036a3120"
I20260619 16:38:53.739281 16597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c7c8b338f1c41229c80bc3022605b7f" candidate_uuid: "d870886ec7ab453d8bc89eba502011c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a0d12ec2e44005857107b48e7adc5f"
I20260619 16:38:53.739389 16459 raft_consensus.cc:3037] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:53.739389 16597 raft_consensus.cc:3037] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:38:53.740592 16459 raft_consensus.cc:2454] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 1.
I20260619 16:38:53.740592 16597 raft_consensus.cc:2454] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d870886ec7ab453d8bc89eba502011c1 in term 1.
I20260619 16:38:53.740911 16663 leader_election.cc:302] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1; no voters:
I20260619 16:38:53.741074 16792 raft_consensus.cc:2781] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:38:53.741355 16792 raft_consensus.cc:686] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 1 LEADER]: Becoming Leader. State: Replica: d870886ec7ab453d8bc89eba502011c1, State: Running, Role: LEADER
I20260619 16:38:53.741679 16792 consensus_queue.cc:237] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:38:53.744621 16316 catalog_manager.cc:5495] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d870886ec7ab453d8bc89eba502011c1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d870886ec7ab453d8bc89eba502011c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } health_report { overall_health: UNKNOWN } } }
I20260619 16:38:53.776443 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.773214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.0.98.3:41419)
0619 16:38:53.773386 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:38:53.773390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:38:53.773423 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:38:53.773489 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:38:53.773491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:38:53.773545 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:38:53.773624 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:38:53.773877 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.774378 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.775161 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:38:53.775380 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:38:53.775472 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:38:53.775702 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:38:53.776119 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:38:53.776221 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:38:53.776289 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 16:38:53.776337 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 16:38:53.782063 16459 raft_consensus.cc:1261] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Refusing update from remote peer d870886ec7ab453d8bc89eba502011c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:53.782071 16597 raft_consensus.cc:1261] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Refusing update from remote peer d870886ec7ab453d8bc89eba502011c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:38:53.782688 16792 consensus_queue.cc:1035] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:53.782886 16796 consensus_queue.cc:1035] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:38:53.790249 16800 mvcc.cc:205] Tried to move back new op lower bound from 7298609699963985920 to 7298609699808514048. Current Snapshot: MvccSnapshot[applied={T|T < 7298609699963985920}]
I20260619 16:38:53.791074 16798 mvcc.cc:205] Tried to move back new op lower bound from 7298609699963985920 to 7298609699808514048. Current Snapshot: MvccSnapshot[applied={T|T < 7298609699963985920 or (T in {7298609699963985920})}]
I20260619 16:38:53.795928 16315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:53.796072 16315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:38:53.798790 16315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Sending DeleteTablet for 3 replicas of tablet 5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:38:53.799474 16439 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7c8b338f1c41229c80bc3022605b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43250
I20260619 16:38:53.799449 16577 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7c8b338f1c41229c80bc3022605b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44066
I20260619 16:38:53.799569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 16355
I20260619 16:38:53.800557 16807 tablet_replica.cc:330] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: stopping tablet replica
I20260619 16:38:53.800541 16709 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c7c8b338f1c41229c80bc3022605b7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:38:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47812
I20260619 16:38:53.800880 16807 raft_consensus.cc:2229] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:38:53.801090 16808 tablet_replica.cc:330] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: stopping tablet replica
I20260619 16:38:53.801187 16807 raft_consensus.cc:2258] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:53.801473 16808 raft_consensus.cc:2229] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:38:53.801891 16808 raft_consensus.cc:2258] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:38:53.802778 16807 ts_tablet_manager.cc:1905] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:53.803375 16808 ts_tablet_manager.cc:1905] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:38:53.806030 16808 ts_tablet_manager.cc:1918] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:53.806180 16808 log.cc:1196] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:38:53.806365 16807 ts_tablet_manager.cc:1918] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:38:53.806555 16807 log.cc:1196] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:38:53.806564 16808 ts_tablet_manager.cc:1939] T 5c7c8b338f1c41229c80bc3022605b7f P d870886ec7ab453d8bc89eba502011c1: Deleting consensus metadata
I20260619 16:38:53.806970 16807 ts_tablet_manager.cc:1939] T 5c7c8b338f1c41229c80bc3022605b7f P 96a0d12ec2e44005857107b48e7adc5f: Deleting consensus metadata
I20260619 16:38:53.807667 16303 catalog_manager.cc:4841] TS d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419): tablet 5c7c8b338f1c41229c80bc3022605b7f (table test-table [id=508c9fc507bf4421938e35e4e274ddf3]) successfully deleted
I20260619 16:38:53.808202 16302 catalog_manager.cc:4841] TS 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505): tablet 5c7c8b338f1c41229c80bc3022605b7f (table test-table [id=508c9fc507bf4421938e35e4e274ddf3]) successfully deleted
W20260619 16:38:53.808349 16303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44443 (error 108)
I20260619 16:38:53.808914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 16509
I20260619 16:38:53.809659 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:38:53.809085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44443 (local address 127.0.0.1:43252)
0619 16:38:53.809524 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:38:53.809595 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345}
W20260619 16:38:53.809993 16303 catalog_manager.cc:4584] TS b5024fe0fcb74bb7a093f787036a3120 (127.0.98.1:44443): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c7c8b338f1c41229c80bc3022605b7f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44443: connect: Connection refused (error 111)
I20260619 16:38:53.817212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 16646
I20260619 16:38:53.822930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 16284
I20260619 16:39:13.829180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--webserver_interface=127.0.98.62
--webserver_port=42541
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:13.938233 16810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:13.938521 16810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:13.938602 16810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:13.942679 16810 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 16:39:13.942756 16810 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 16:39:13.942785 16810 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 16:39:13.942826 16810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:13.942862 16810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:13.942920 16810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:13.942961 16810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:13.943007 16810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:13.948190 16810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:13.949215 16810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:13.950235 16810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:39:13.959271 16810 server_base.cc:1034] running on GCE node
W20260619 16:39:13.959182 16815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:13.959178 16816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:13.959165 16818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:13.960090 16810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:13.961256 16810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:13.962473 16810 hybrid_clock.cc:648] HybridClock initialized: now 1781887153962415 us; error 73 us; skew 500 ppm
Jun 19 16:39:13 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:13.971612 16810 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:39:13.972529 16810 webserver.cc:460] Webserver started at http://127.0.98.62:42541/ using document root <none> and password file <none>
I20260619 16:39:13.973096 16810 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:13.973194 16810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:13.976986 16810 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:13.979238 16825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:13.980446 16810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:13.980609 16810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "7ec1beef6b2d493d9183ce0ff0332260"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "d0937e9d610344bc3ac30647ab03a767"
server_key_iv: "6f5c23cb210050d8c853aa748558c113"
server_key_version: "encryptionkey@0"
I20260619 16:39:13.981206 16810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:14.004278 16810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:14.007511 16810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:14.007746 16810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:14.015408 16810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36777
I20260619 16:39:14.015731 16877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36777 every 8 connection(s)
I20260619 16:39:14.016528 16810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:14.020794 16878 sys_catalog.cc:263] Verifying existing consensus state
I20260619 16:39:14.022383 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap starting.
I20260619 16:39:14.025667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16810
I20260619 16:39:14.026113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44443
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36777
--webserver_port=41187
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:39:14.039646 16878 log.cc:824] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:14.051110 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:39:14.051496 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Bootstrap complete.
I20260619 16:39:14.055459 16878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:39:14.055851 16878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec1beef6b2d493d9183ce0ff0332260, State: Initialized, Role: FOLLOWER
I20260619 16:39:14.056384 16878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:39:14.056541 16878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:14.056628 16878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:14.056747 16878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 2 FOLLOWER]: Advancing to term 3
I20260619 16:39:14.058645 16878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:39:14.058974 16878 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec1beef6b2d493d9183ce0ff0332260; no voters:
I20260619 16:39:14.059293 16878 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 16:39:14.059501 16882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 16:39:14.059752 16882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [term 3 LEADER]: Becoming Leader. State: Replica: 7ec1beef6b2d493d9183ce0ff0332260, State: Running, Role: LEADER
I20260619 16:39:14.060215 16882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } }
I20260619 16:39:14.060554 16878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:14.061228 16882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec1beef6b2d493d9183ce0ff0332260. Latest consensus state: current_term: 3 leader_uuid: "7ec1beef6b2d493d9183ce0ff0332260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260619 16:39:14.061460 16882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:14.061667 16883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7ec1beef6b2d493d9183ce0ff0332260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec1beef6b2d493d9183ce0ff0332260" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36777 } } }
I20260619 16:39:14.061784 16883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:14.062026 16890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:14.067142 16890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:14.067797 16890 catalog_manager.cc:1259] Loaded cluster ID: 6d8690df104c4981a2ba1dd41d35558c
I20260619 16:39:14.067859 16890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:14.070269 16890 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:14.071071 16890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Loaded TSK: 0
I20260619 16:39:14.071172 16890 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7ec1beef6b2d493d9183ce0ff0332260: Loaded TSK: 1
F20260619 16:39:14.072085 16890 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f61a87f69ad google::LogMessage::Fail() at ??:0
@ 0x7f61a87f988a google::LogMessage::SendToLog() at ??:0
@ 0x7f61a87f64de google::LogMessage::Flush() at ??:0
@ 0x7f61a87f7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f61ab4ba18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f61ab4b2f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f61b315b7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f61b313599c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f61b3136178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f61b3177df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f61b319682b std::function<>::operator()() at ??:0
@ 0x7f61b3135b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f61b3136f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f61b3133579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f61b3177640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f61aaa483d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f61aaa48c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f61aaa4a3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f61aaa39512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f61ac07d6db start_thread at ??:0
@ 0x7f61a72f461f clone at ??:0
W20260619 16:39:14.173099 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:14.173395 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:14.173482 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:14.177539 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:14.177595 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:14.177681 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:14.183012 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41187
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:14.184041 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:14.185021 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:14.194228 16926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:14.194283 16925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:14.194231 16928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:14.194763 16880 server_base.cc:1034] running on GCE node
I20260619 16:39:14.195211 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:14.195997 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:14.197192 16880 hybrid_clock.cc:648] HybridClock initialized: now 1781887154197165 us; error 38 us; skew 500 ppm
Jun 19 16:39:14 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:14.200515 16880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:39:14.201444 16880 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260619 16:39:14.201978 16880 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:14.202073 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:14.206001 16880 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260619 16:39:14.208132 16935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.209447 16880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:14.209576 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "b5024fe0fcb74bb7a093f787036a3120"
format_stamp: "Formatted at 2026-06-19 16:38:31 on dist-test-slave-0f28"
server_key: "8bf6b7eb686eac3bd112391c149a3ddf"
server_key_iv: "7cac85fdaba07b359552ab2b66e3084e"
server_key_version: "encryptionkey@0"
I20260619 16:39:14.210086 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:14.228262 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:14.231410 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:14.231598 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:14.232141 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:14.233222 16942 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 16:39:14.238822 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 16:39:14.238904 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260619 16:39:14.238981 16880 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 16:39:14.240540 16880 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 16:39:14.240607 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 16:39:14.240936 16942 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Bootstrap starting.
I20260619 16:39:14.251024 16880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44443
I20260619 16:39:14.251240 17049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44443 every 8 connection(s)
I20260619 16:39:14.251966 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:14.252936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 16880
I20260619 16:39:14.253434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38505
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36777
--webserver_port=43707
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:39:14.265141 16942 log.cc:824] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:14.273883 16942 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 16:39:14.274297 16942 tablet_bootstrap.cc:492] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Bootstrap complete.
I20260619 16:39:14.275691 16942 ts_tablet_manager.cc:1397] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.010s
I20260619 16:39:14.278940 16942 raft_consensus.cc:348] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:39:14.279482 16942 raft_consensus.cc:729] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5024fe0fcb74bb7a093f787036a3120, State: Initialized, Role: FOLLOWER
I20260619 16:39:14.280004 16942 consensus_queue.cc:260] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:39:14.281005 16942 ts_tablet_manager.cc:1428] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260619 16:39:14.299206 17053 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:39:14.254982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:39347)
0619 16:39:14.255804 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.255860 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:14.256876 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:14.298444 (+ 41568us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":466,"threads_started":1}
W20260619 16:39:14.299693 17050 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36777 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36777: Client connection negotiation failed: client connection to 127.0.98.62:36777: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 16:39:14.300393 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.300142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:53837)
0619 16:39:14.300270 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.300342 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 16:39:14.301000 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.300806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:43589)
0619 16:39:14.300903 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.300945 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:39:14.301157 17050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 16:39:14.371392 17052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:14.371660 17052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:14.371735 17052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:14.375659 17052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:14.375744 17052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:14.375840 17052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:14.380937 17052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43707
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:14.381913 17052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:14.382863 17052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:14.392071 17061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:14.392081 17060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:14.392081 17063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:14.392474 17052 server_base.cc:1034] running on GCE node
I20260619 16:39:14.392865 17052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:14.393468 17052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:14.394627 17052 hybrid_clock.cc:648] HybridClock initialized: now 1781887154394608 us; error 30 us; skew 500 ppm
Jun 19 16:39:14 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:14.397303 17052 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:39:14.398166 17052 webserver.cc:460] Webserver started at http://127.0.98.2:43707/ using document root <none> and password file <none>
I20260619 16:39:14.398651 17052 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:14.398710 17052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:14.402396 17052 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 16:39:14.404407 17070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.405661 17052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:14.405803 17052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "96a0d12ec2e44005857107b48e7adc5f"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "32f3064db3ba92c2fe26b3c1e0a490e1"
server_key_iv: "242483b770f04d2947258cff3a52c9b7"
server_key_version: "encryptionkey@0"
I20260619 16:39:14.406375 17052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:14.421057 17052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:14.424458 17052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:14.424669 17052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:14.425240 17052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:14.426088 17052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:14.426160 17052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.426225 17052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:14.426277 17052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.435331 17052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38505
I20260619 16:39:14.435364 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38505 every 8 connection(s)
I20260619 16:39:14.436262 17052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:39:14.439846 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.438377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:44287)
0619 16:39:14.439046 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.439173 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":126,"threads_started":1}
W20260619 16:39:14.440289 17184 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36777 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36777: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
I20260619 16:39:14.440871 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.440630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:60273)
0619 16:39:14.440766 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.440814 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:39:14.441385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17052
I20260619 16:39:14.441571 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.441337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:42065)
0619 16:39:14.441458 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.441514 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:39:14.441789 17184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:39:14.441969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41419
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36777
--webserver_port=33927
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:14.550446 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:14.550712 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:14.550786 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:14.554677 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:14.554759 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:14.554852 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:14.559943 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33927
--tserver_master_addrs=127.0.98.62:36777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:14.560875 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:14.561831 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:14.570904 17195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:14.570999 17187 server_base.cc:1034] running on GCE node
W20260619 16:39:14.570963 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:14.570904 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:14.571477 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:14.572078 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:14.573242 17187 hybrid_clock.cc:648] HybridClock initialized: now 1781887154573213 us; error 50 us; skew 500 ppm
Jun 19 16:39:14 dist-test-slave-0f28 krb5kdc[15737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:14.575917 17187 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:39:14.576719 17187 webserver.cc:460] Webserver started at http://127.0.98.3:33927/ using document root <none> and password file <none>
I20260619 16:39:14.577258 17187 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:14.577360 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:14.580916 17187 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 16:39:14.582978 17202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.584149 17187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:14.584280 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "d870886ec7ab453d8bc89eba502011c1"
format_stamp: "Formatted at 2026-06-19 16:38:32 on dist-test-slave-0f28"
server_key: "8e809a41cf8c0a99d9b54c7633e94e47"
server_key_iv: "ae206b7306b012ceff0b168775a3ab6a"
server_key_version: "encryptionkey@0"
I20260619 16:39:14.584781 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:14.602981 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:14.606209 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:14.606412 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:14.607008 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:14.607842 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:14.607924 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.608016 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:14.608068 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:14.617442 17187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41419
I20260619 16:39:14.617497 17315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41419 every 8 connection(s)
I20260619 16:39:14.618629 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:14.622561 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.620676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:50999)
0619 16:39:14.621182 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.621838 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
W20260619 16:39:14.623054 17316 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36777 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36777: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
I20260619 16:39:14.623648 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.623418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:56613)
0619 16:39:14.623535 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.623585 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 16:39:14.624392 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:14.624189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:43245)
0619 16:39:14.624300 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:14.624344 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:39:14.624673 17316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 16:39:14.628043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17187
I20260619 16:39:15.302192 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.301755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:48511)
0619 16:39:15.302042 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:15.302096 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
I20260619 16:39:15.442643 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.442232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:56803)
0619 16:39:15.442501 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:15.442549 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":65,"threads_started":1}
I20260619 16:39:15.625655 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.625158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:55233)
0619 16:39:15.625511 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:15.625562 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":71,"threads_started":1}
I20260619 16:39:15.730275 17323 raft_consensus.cc:482] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:15.730517 17323 raft_consensus.cc:504] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:39:15.731673 17323 leader_election.cc:288] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505), d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419)
Jun 19 16:39:15 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 16:39:15 dist-test-slave-0f28 krb5kdc[15737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 16:39:15.745642 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.731912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41419 (local address 127.0.98.1:35293)
0619 16:39:15.732083 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:15.732101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:15.732206 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:15.734046 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:15.734052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:15.734584 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:15.735021 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:15.735034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.736272 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.736276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:15.737119 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:15.737129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.737859 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.737863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:15.738045 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:15.738760 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:15.738775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:15.740710 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:15.742740 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:15.742743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:15.742745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:15.743139 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:15.743582 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:15.743588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:15.743595 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:15.743818 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:15.744538 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:15.744543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:15.744878 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 16:39:15.745097 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 16:39:15.745450 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":75,"spinlock_wait_cycles":13056}
I20260619 16:39:15.745746 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.732156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47257 (local address 127.0.98.2:38505)
0619 16:39:15.732558 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:15.732579 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:15.732628 (+ 49us) server_negotiation.cc:407] Connection header received
0619 16:39:15.733284 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:15.733290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:15.733713 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:15.734009 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:15.735193 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.736132 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.737247 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.737748 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.737923 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:15.740561 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:15.740566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:15.740576 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:15.740603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:15.742526 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:15.743277 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:15.743283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:15.743288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:15.743438 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:15.744084 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:15.744088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:15.744089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:15.744498 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:15.744598 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:15.745241 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 16:39:15.745562 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":111,"threads_started":1}
I20260619 16:39:15.746023 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.731922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38505 (local address 127.0.98.1:47257)
0619 16:39:15.732435 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:15.732448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:15.732530 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:15.734047 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:15.734053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:15.734585 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:15.735021 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:15.735035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.736291 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.736294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:15.737122 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:15.737131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.737872 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.737876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:15.738039 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:15.738763 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:15.738780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:15.740313 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:15.742655 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:15.742663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:15.742678 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:15.743144 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:15.743930 (+ 786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:15.743932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:15.743933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:15.743998 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:15.744601 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:15.744604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:15.744878 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 16:39:15.745097 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 16:39:15.745856 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":705,"spinlock_wait_cycles":26240,"thread_start_us":72,"threads_started":1}
I20260619 16:39:15.746034 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:15.732317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35293 (local address 127.0.98.3:41419)
0619 16:39:15.732724 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:15.732742 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:15.732771 (+ 29us) server_negotiation.cc:407] Connection header received
0619 16:39:15.733434 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:15.733440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:15.733715 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:15.734000 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:15.735200 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.736138 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.737243 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:15.737741 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:15.737906 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:15.740889 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:15.740894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:15.740904 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:15.740928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:15.742553 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:15.743277 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:15.743283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:15.743288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:15.743437 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:15.743921 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:15.743923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:15.743924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:15.744420 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:15.744552 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:15.745241 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 16:39:15.745699 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":88,"threads_started":1}
W20260619 16:39:15.748433 16937 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
W20260619 16:39:15.748674 16938 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:39:15.748746 16938 leader_election.cc:302] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5024fe0fcb74bb7a093f787036a3120; no voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1
I20260619 16:39:15.749043 17323 raft_consensus.cc:2726] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:39:16.303480 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:16.303068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:48621)
0619 16:39:16.303329 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:16.303380 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":74,"threads_started":1}
I20260619 16:39:16.443786 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:16.443346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:60067)
0619 16:39:16.443642 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:16.443697 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260619 16:39:16.626878 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:16.626413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:47391)
0619 16:39:16.626728 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:16.626788 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":83,"threads_started":1}
I20260619 16:39:17.304582 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:17.304146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:48015)
0619 16:39:17.304432 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:17.304488 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":83,"threads_started":1}
I20260619 16:39:17.422080 17331 raft_consensus.cc:482] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:17.422254 17331 raft_consensus.cc:504] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:39:17.422713 17331 leader_election.cc:288] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505), d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419)
W20260619 16:39:17.423674 16937 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
W20260619 16:39:17.423781 16938 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:39:17.423900 16938 leader_election.cc:302] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5024fe0fcb74bb7a093f787036a3120; no voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1
I20260619 16:39:17.424057 17331 raft_consensus.cc:2726] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:39:17.444938 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:17.444499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:34865)
0619 16:39:17.444796 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:17.444849 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":70,"threads_started":1}
I20260619 16:39:17.628067 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:17.627677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:37349)
0619 16:39:17.627912 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:17.627964 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260619 16:39:18.305925 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:18.305498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:42631)
0619 16:39:18.305770 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:18.305829 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260619 16:39:18.446239 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:18.445732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:47145)
0619 16:39:18.446075 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:18.446134 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":81,"threads_started":1}
I20260619 16:39:18.629202 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:18.628805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:42181)
0619 16:39:18.629058 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:18.629113 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260619 16:39:19.307341 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:19.306901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:46159)
0619 16:39:19.307186 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:19.307244 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":74,"threads_started":1}
I20260619 16:39:19.447440 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:19.447038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:35165)
0619 16:39:19.447287 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:19.447341 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260619 16:39:19.630524 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:19.630054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:55323)
0619 16:39:19.630329 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:19.630399 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260619 16:39:20.223847 17340 raft_consensus.cc:482] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:20.223982 17340 raft_consensus.cc:504] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:39:20.224490 17340 leader_election.cc:288] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505), d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419)
W20260619 16:39:20.225380 16938 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
W20260619 16:39:20.225531 16937 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:39:20.225641 16937 leader_election.cc:302] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5024fe0fcb74bb7a093f787036a3120; no voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1
I20260619 16:39:20.225807 17340 raft_consensus.cc:2726] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:39:20.308552 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:20.308145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:52927)
0619 16:39:20.308404 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:20.308459 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260619 16:39:20.448529 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:20.448175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:53537)
0619 16:39:20.448395 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:20.448443 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":71,"threads_started":1}
I20260619 16:39:20.631793 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:20.631333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:57533)
0619 16:39:20.631593 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:20.631698 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260619 16:39:21.309922 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:21.309458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:43045)
0619 16:39:21.309757 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:21.309823 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":84,"threads_started":1}
I20260619 16:39:21.449843 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:21.449416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:48759)
0619 16:39:21.449698 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:21.449753 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":67,"threads_started":1}
I20260619 16:39:21.633018 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:21.632560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:49637)
0619 16:39:21.632872 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:21.632926 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":81,"threads_started":1}
I20260619 16:39:22.311355 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:22.310885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:42541)
0619 16:39:22.311202 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:22.311262 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":134,"threads_started":1}
I20260619 16:39:22.451064 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:22.450682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:37757)
0619 16:39:22.450920 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:22.450973 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260619 16:39:22.634260 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:22.633838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:46461)
0619 16:39:22.634105 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:22.634162 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260619 16:39:23.312623 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:23.312177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:44767)
0619 16:39:23.312454 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:23.312514 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260619 16:39:23.452284 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:23.451826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:50091)
0619 16:39:23.452111 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:23.452184 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260619 16:39:23.635421 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:23.634975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:36365)
0619 16:39:23.635260 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:23.635322 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260619 16:39:23.977536 17353 raft_consensus.cc:482] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:23.977675 17353 raft_consensus.cc:504] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a0d12ec2e44005857107b48e7adc5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38505 } } peers { permanent_uuid: "d870886ec7ab453d8bc89eba502011c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41419 } } peers { permanent_uuid: "b5024fe0fcb74bb7a093f787036a3120" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44443 } }
I20260619 16:39:23.978120 17353 leader_election.cc:288] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505), d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419)
W20260619 16:39:23.978967 16938 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d870886ec7ab453d8bc89eba502011c1 (127.0.98.3:41419): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
W20260619 16:39:23.979074 16937 leader_election.cc:341] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 96a0d12ec2e44005857107b48e7adc5f (127.0.98.2:38505): Not found: Tablet not found: 5c7c8b338f1c41229c80bc3022605b7f
I20260619 16:39:23.979204 16937 leader_election.cc:302] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5024fe0fcb74bb7a093f787036a3120; no voters: 96a0d12ec2e44005857107b48e7adc5f, d870886ec7ab453d8bc89eba502011c1
I20260619 16:39:23.979375 17353 raft_consensus.cc:2726] T 5c7c8b338f1c41229c80bc3022605b7f P b5024fe0fcb74bb7a093f787036a3120 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 16:39:24.313977 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:24.313546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:55883)
0619 16:39:24.313797 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:24.313853 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260619 16:39:24.453748 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:24.453230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:43989)
0619 16:39:24.453592 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:24.453655 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260619 16:39:24.636858 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:24.636305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:41699)
0619 16:39:24.636632 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:24.636716 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":78,"threads_started":1}
I20260619 16:39:25.315237 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:25.314801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:34751)
0619 16:39:25.315070 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:25.315139 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260619 16:39:25.454969 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:25.454551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:50649)
0619 16:39:25.454812 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:25.454871 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":69,"threads_started":1}
I20260619 16:39:25.638128 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:25.637736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:35759)
0619 16:39:25.637984 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:25.638037 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260619 16:39:26.316617 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:26.316197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:33855)
0619 16:39:26.316465 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:26.316524 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":114,"threads_started":1}
I20260619 16:39:26.456207 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:26.455784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:37805)
0619 16:39:26.456039 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:26.456098 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260619 16:39:26.639369 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:26.638934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:48643)
0619 16:39:26.639197 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:26.639275 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":91,"threads_started":1}
I20260619 16:39:27.317834 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:27.317414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:38113)
0619 16:39:27.317683 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:27.317739 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":71,"threads_started":1}
I20260619 16:39:27.457476 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:27.456995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:37755)
0619 16:39:27.457269 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:27.457382 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260619 16:39:27.640542 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:27.640116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:35197)
0619 16:39:27.640392 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:27.640449 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260619 16:39:28.319061 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:28.318602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:39065)
0619 16:39:28.318889 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:28.318963 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260619 16:39:28.458642 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:28.458208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:53995)
0619 16:39:28.458474 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:28.458528 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260619 16:39:28.641827 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:28.641385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.3:50515)
0619 16:39:28.641649 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:28.641735 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260619 16:39:29.320473 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:29.320051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.1:56643)
0619 16:39:29.320315 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:29.320378 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
I20260619 16:39:29.460041 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:29.459647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36777 (local address 127.0.98.2:48861)
0619 16:39:29.459887 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:29.459938 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":67,"threads_started":1}
I20260619 16:39:29.630990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 16880
I20260619 16:39:29.636915 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17052
I20260619 16:39:29.642056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17187
2026-06-19T16:39:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59359 ms)
[----------] 30 tests from SecurityITest (242555 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:29 dist-test-slave-0f28 krb5kdc[17373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:29 dist-test-slave-0f28 krb5kdc[17373](info): set up 2 sockets
Jun 19 16:39:29 dist-test-slave-0f28 krb5kdc[17373](info): commencing operation
krb5kdc: starting...
W20260619 16:39:30.693651 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:30 dist-test-slave-0f28 krb5kdc[17373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:39:30.821058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:30.935421 17386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:30.935771 17386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:30.935896 17386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:30.940304 17386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:30.940371 17386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:30.940392 17386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:30.940409 17386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:30.940425 17386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:30.945778 17386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:30.946796 17386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:30.947781 17386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:30.956338 17392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:30.956350 17394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:30.956377 17391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:30.956421 17386 server_base.cc:1034] running on GCE node
I20260619 16:39:30.957080 17386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:30.958185 17386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:30.959424 17386 hybrid_clock.cc:648] HybridClock initialized: now 1781887170959396 us; error 40 us; skew 500 ppm
Jun 19 16:39:30 dist-test-slave-0f28 krb5kdc[17373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:30.962193 17386 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:39:30.963083 17386 webserver.cc:460] Webserver started at http://127.0.98.62:33761/ using document root <none> and password file <none>
I20260619 16:39:30.963640 17386 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:30.963730 17386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:30.963963 17386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:30.966043 17386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "e50da62dc31346dfbd3a5b5d47bd857c"
format_stamp: "Formatted at 2026-06-19 16:39:30 on dist-test-slave-0f28"
server_key: "7e060ffec40bb0aa71c072593d18ec41"
server_key_iv: "4cda5cae580f669bef2c96cad34cf4b1"
server_key_version: "encryptionkey@0"
I20260619 16:39:30.966596 17386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "e50da62dc31346dfbd3a5b5d47bd857c"
format_stamp: "Formatted at 2026-06-19 16:39:30 on dist-test-slave-0f28"
server_key: "7e060ffec40bb0aa71c072593d18ec41"
server_key_iv: "4cda5cae580f669bef2c96cad34cf4b1"
server_key_version: "encryptionkey@0"
I20260619 16:39:30.970129 17386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:30.972416 17401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:30.973685 17386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:30.973832 17386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "e50da62dc31346dfbd3a5b5d47bd857c"
format_stamp: "Formatted at 2026-06-19 16:39:30 on dist-test-slave-0f28"
server_key: "7e060ffec40bb0aa71c072593d18ec41"
server_key_iv: "4cda5cae580f669bef2c96cad34cf4b1"
server_key_version: "encryptionkey@0"
I20260619 16:39:30.973948 17386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:30.993996 17386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:30.997028 17386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:30.997264 17386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:31.004272 17386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33917
I20260619 16:39:31.004294 17453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33917 every 8 connection(s)
I20260619 16:39:31.005216 17386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:31.007181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17386
I20260619 16:39:31.007316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:31.007632 392 external_mini_cluster.cc:1428] Setting key 542c25d4ee219a805bea58731732c66b
I20260619 16:39:31.008407 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:31.014361 17454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c: Bootstrap starting.
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:31.016830 17454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:31.017645 17454 log.cc:824] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:31.019483 17454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c: No bootstrap required, opened a new log
I20260619 16:39:31.021680 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.009086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.0.98.62:33917)
0619 16:39:31.009582 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.009598 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.009631 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:39:31.010262 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.010277 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.010591 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.010957 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:31.011752 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.012762 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.013711 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.014187 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.014360 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.016983 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:31.016988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:31.017001 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:31.017027 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:31.018877 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.019350 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.019362 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.019367 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.019472 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.019778 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.019781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.019782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.020157 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:31.020367 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.020648 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.020926 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":175,"threads_started":1}
I20260619 16:39:31.023108 17454 raft_consensus.cc:348] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } }
I20260619 16:39:31.023353 17454 raft_consensus.cc:374] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:31.023432 17454 raft_consensus.cc:729] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50da62dc31346dfbd3a5b5d47bd857c, State: Initialized, Role: FOLLOWER
I20260619 16:39:31.023859 17454 consensus_queue.cc:260] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } }
I20260619 16:39:31.023979 17454 raft_consensus.cc:388] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:31.024060 17454 raft_consensus.cc:482] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:31.024159 17454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:31.025009 17454 raft_consensus.cc:504] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } }
I20260619 16:39:31.025377 17454 leader_election.cc:302] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e50da62dc31346dfbd3a5b5d47bd857c; no voters:
I20260619 16:39:31.025681 17454 leader_election.cc:288] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:31.025935 17459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:31.026180 17459 raft_consensus.cc:686] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [term 1 LEADER]: Becoming Leader. State: Replica: e50da62dc31346dfbd3a5b5d47bd857c, State: Running, Role: LEADER
I20260619 16:39:31.026504 17459 consensus_queue.cc:237] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } }
I20260619 16:39:31.026902 17454 sys_catalog.cc:564] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:31.027617 17459 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e50da62dc31346dfbd3a5b5d47bd857c. Latest consensus state: current_term: 1 leader_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } }
I20260619 16:39:31.027736 17459 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:31.028241 17468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:31.028188 17460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50da62dc31346dfbd3a5b5d47bd857c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33917 } } }
I20260619 16:39:31.028299 17460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:31.031036 17468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:31.036326 17468 catalog_manager.cc:1347] Generated new cluster ID: c8e198769dca435e847096847fea167e
I20260619 16:39:31.036399 17468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:31.050551 17468 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:31.051446 17468 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:31.058888 17468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c: Generated new TSK 0
I20260619 16:39:31.059525 17468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:39:31.095961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33917
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:31.232829 17480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:31.233103 17480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:31.233150 17480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:31.237527 17480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:31.237589 17480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:31.237653 17480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:31.243047 17480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:31.243948 17480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:31.244851 17480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:31.254441 17488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:31.254457 17485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:31.254463 17486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:31.254793 17480 server_base.cc:1034] running on GCE node
I20260619 16:39:31.255178 17480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:31.255858 17480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:31.257047 17480 hybrid_clock.cc:648] HybridClock initialized: now 1781887171257025 us; error 39 us; skew 500 ppm
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:31.259938 17480 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:39:31.260818 17480 webserver.cc:460] Webserver started at http://127.0.98.1:43403/ using document root <none> and password file <none>
I20260619 16:39:31.261384 17480 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:31.261448 17480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:31.261613 17480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:31.263621 17480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "07552443907e4eef9536e60834f5f117"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "1d70c61f500823de3c224b6d89e61ea6"
server_key_iv: "8709afba30b2b868ce058ec4229b1c28"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.264181 17480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "07552443907e4eef9536e60834f5f117"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "1d70c61f500823de3c224b6d89e61ea6"
server_key_iv: "8709afba30b2b868ce058ec4229b1c28"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.267788 17480 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:31.270256 17495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.271644 17480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:39:31.271795 17480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "07552443907e4eef9536e60834f5f117"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "1d70c61f500823de3c224b6d89e61ea6"
server_key_iv: "8709afba30b2b868ce058ec4229b1c28"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.271912 17480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:31.289240 17480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:31.292232 17480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:31.292441 17480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:31.293073 17480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:31.293931 17480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:31.293983 17480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.294052 17480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:31.294082 17480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.303241 17480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39055
I20260619 16:39:31.303294 17608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39055 every 8 connection(s)
I20260619 16:39:31.304095 17480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:31.313138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17480
I20260619 16:39:31.313299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:31.313683 392 external_mini_cluster.cc:1428] Setting key 375aec357a2209f416086147a3cc348c
I20260619 16:39:31.317960 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.306042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52697 (local address 127.0.98.62:33917)
0619 16:39:31.306188 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.306192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.306966 (+ 774us) server_negotiation.cc:407] Connection header received
0619 16:39:31.308234 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.308238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.308294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.308398 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:31.309530 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.310079 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.311198 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.311439 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.311553 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.314117 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:31.314120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:31.314123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:31.314173 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:31.315805 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.316312 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.316317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.316319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.316409 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.316826 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.316828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.316830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.317001 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:31.317086 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.317743 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.317848 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 16:39:31.319049 17611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.306400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33917 (local address 127.0.98.1:52697)
0619 16:39:31.306818 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.306857 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.308019 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.308560 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.308566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.308904 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:31.309370 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.309385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.310206 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.310210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.311067 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.311076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.311586 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.311590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.311758 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.312283 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:31.312298 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:31.313933 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:31.315940 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:31.315945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:31.315958 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:31.316206 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:31.316510 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:31.316515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:31.316518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:31.316721 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:31.317110 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:31.317119 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:31.317446 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.317670 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.318013 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
I20260619 16:39:31.320279 17609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33917
I20260619 16:39:31.320513 17609 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:31.320974 17609 heartbeater.cc:511] Master 127.0.98.62:33917 requested a full tablet report, sending...
I20260619 16:39:31.322589 17418 ts_manager.cc:194] Registered new tserver with Master: 07552443907e4eef9536e60834f5f117 (127.0.98.1:39055)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:39:31.324007 17418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:39:31.344321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33917
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:31.452195 17614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:31.452425 17614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:31.452466 17614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:31.456498 17614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:31.456552 17614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:31.456667 17614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:31.462052 17614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:31.463047 17614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:31.464005 17614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:31.473544 17620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:31.473642 17622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:31.473552 17619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:31.473806 17614 server_base.cc:1034] running on GCE node
I20260619 16:39:31.474195 17614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:31.474741 17614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:31.475946 17614 hybrid_clock.cc:648] HybridClock initialized: now 1781887171475932 us; error 32 us; skew 500 ppm
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:31.479141 17614 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:39:31.480077 17614 webserver.cc:460] Webserver started at http://127.0.98.2:46527/ using document root <none> and password file <none>
I20260619 16:39:31.480635 17614 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:31.480767 17614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:31.480995 17614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:31.483065 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "91e1e023d3f94839b558a1b742b00999"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "aa3d7490e61cf022139a77d8ae39292c"
server_key_iv: "f712da10a51a449b7fcd666cf3ae0f1b"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.483632 17614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "91e1e023d3f94839b558a1b742b00999"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "aa3d7490e61cf022139a77d8ae39292c"
server_key_iv: "f712da10a51a449b7fcd666cf3ae0f1b"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.487525 17614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:39:31.490010 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.491231 17614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:31.491374 17614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "91e1e023d3f94839b558a1b742b00999"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "aa3d7490e61cf022139a77d8ae39292c"
server_key_iv: "f712da10a51a449b7fcd666cf3ae0f1b"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.491521 17614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:31.510880 17614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:31.513764 17614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:31.513993 17614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:31.514576 17614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:31.515383 17614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:31.515455 17614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.515532 17614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:31.515576 17614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.524641 17614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44659
I20260619 16:39:31.524660 17742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44659 every 8 connection(s)
I20260619 16:39:31.525552 17614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:39:31.530299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17614
I20260619 16:39:31.530390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:39:31.530697 392 external_mini_cluster.cc:1428] Setting key 80175ebacc36da0839b05df284130306
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:31.539623 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.527354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40831 (local address 127.0.98.62:33917)
0619 16:39:31.527510 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.527513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.528456 (+ 943us) server_negotiation.cc:407] Connection header received
0619 16:39:31.529617 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.529620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.529675 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.529758 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:31.531030 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.531705 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.532929 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.533146 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.533266 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.535739 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:31.535743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:31.535746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:31.535777 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:31.537567 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.538068 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.538073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.538075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.538169 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.538482 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.538489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.538491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.538734 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:31.538843 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.539292 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.539417 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:39:31.540328 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.527714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33917 (local address 127.0.98.2:40831)
0619 16:39:31.528264 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.528328 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.529406 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.529904 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.529911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.530248 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:31.530883 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.530899 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.531835 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.531838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.532806 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.532816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.533301 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.533305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.533490 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.534067 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:31.534082 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:31.535566 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:31.537681 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:31.537686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:31.537721 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:31.537967 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:31.538267 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:31.538270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:31.538271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:31.538391 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:31.538839 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:31.538844 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:31.539058 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.539205 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.539433 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":149,"threads_started":1}
I20260619 16:39:31.541473 17743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33917
I20260619 16:39:31.541708 17743 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:31.542222 17743 heartbeater.cc:511] Master 127.0.98.62:33917 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:39:31.543349 17418 ts_manager.cc:194] Registered new tserver with Master: 91e1e023d3f94839b558a1b742b00999 (127.0.98.2:44659)
I20260619 16:39:31.543920 17418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:39:31.561219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33917
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:31.669747 17748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:31.669983 17748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:31.670055 17748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:31.674160 17748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:31.674220 17748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:31.674309 17748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:31.679870 17748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:31.680840 17748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:31.681876 17748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:31.691172 17754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:31.691172 17753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:31.691315 17756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:31.691543 17748 server_base.cc:1034] running on GCE node
I20260619 16:39:31.691967 17748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:31.692612 17748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:31.693822 17748 hybrid_clock.cc:648] HybridClock initialized: now 1781887171693810 us; error 25 us; skew 500 ppm
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:31.696643 17748 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:39:31.697551 17748 webserver.cc:460] Webserver started at http://127.0.98.3:32805/ using document root <none> and password file <none>
I20260619 16:39:31.698093 17748 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:31.698180 17748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:31.698402 17748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:31.700374 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "15c5ad038fe8464a91aedc02b75624c3"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "123d5b9e1b2fb0bb4f2f58d3f68e79d8"
server_key_iv: "6d074dc6cd24b23a3cdc4e120da1577f"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.700928 17748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15c5ad038fe8464a91aedc02b75624c3"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "123d5b9e1b2fb0bb4f2f58d3f68e79d8"
server_key_iv: "6d074dc6cd24b23a3cdc4e120da1577f"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.704473 17748 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 16:39:31.706853 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.707986 17748 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:31.708122 17748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "15c5ad038fe8464a91aedc02b75624c3"
format_stamp: "Formatted at 2026-06-19 16:39:31 on dist-test-slave-0f28"
server_key: "123d5b9e1b2fb0bb4f2f58d3f68e79d8"
server_key_iv: "6d074dc6cd24b23a3cdc4e120da1577f"
server_key_version: "encryptionkey@0"
I20260619 16:39:31.708230 17748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:31.725718 17748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:31.729367 17748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:31.729585 17748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:31.730186 17748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:31.731014 17748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:31.731086 17748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.731145 17748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:31.731197 17748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:31.739888 17748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43175
I20260619 16:39:31.739910 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43175 every 8 connection(s)
I20260619 16:39:31.740736 17748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:31.747134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17748
I20260619 16:39:31.747246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:39:31.747538 392 external_mini_cluster.cc:1428] Setting key 381771b431059a91650572f9dca453f2
Jun 19 16:39:31 dist-test-slave-0f28 krb5kdc[17373](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:31.754827 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.742520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37247 (local address 127.0.98.62:33917)
0619 16:39:31.742730 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.742734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.743689 (+ 955us) server_negotiation.cc:407] Connection header received
0619 16:39:31.744773 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.744776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.744821 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.744894 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:31.746078 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.746623 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.747950 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.748207 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.748289 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.750985 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:31.750988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:31.750990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:31.751012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:31.752493 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.753074 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.753078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.753080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.753222 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.753647 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.753651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.753653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.753934 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:31.754030 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.754522 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.754653 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260619 16:39:31.755539 17879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.742864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33917 (local address 127.0.98.3:37247)
0619 16:39:31.743530 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.743603 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.744595 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.745043 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.745049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.745450 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:31.745937 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.745950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.746750 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.746753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.747830 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.747840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.748444 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.748447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.748581 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.749203 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:31.749218 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:31.750828 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:31.752623 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:31.752628 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:31.752678 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:31.752925 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:31.753359 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:31.753362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:31.753364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:31.753528 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:31.754038 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:31.754047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:31.754285 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.754464 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.754768 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":135,"threads_started":1}
I20260619 16:39:31.756590 17877 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33917
I20260619 16:39:31.756803 17877 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:31.757232 17877 heartbeater.cc:511] Master 127.0.98.62:33917 requested a full tablet report, sending...
I20260619 16:39:31.758273 17418 ts_manager.cc:194] Registered new tserver with Master: 15c5ad038fe8464a91aedc02b75624c3 (127.0.98.3:43175)
I20260619 16:39:31.758833 17418 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37247
I20260619 16:39:31.761564 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:39:31.770651 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.763467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.0.98.62:33917)
0619 16:39:31.763601 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.763606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.763836 (+ 230us) server_negotiation.cc:407] Connection header received
0619 16:39:31.763922 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.763924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.763960 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.764024 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:31.764828 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.765312 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.766240 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.766442 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.766523 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.767884 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:31.767887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:31.767889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:31.767909 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:31.769158 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.769587 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.769589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.769591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.769685 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:31.769936 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:31.769938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:31.769939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:31.770129 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:31.770229 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.770390 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.770472 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:39:31.773267 17418 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:39:31.776531 17418 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:39:31.790398 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.784745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.0.98.2:44659)
0619 16:39:31.785124 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.785138 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.785161 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:39:31.785261 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.785267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.785495 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.785678 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:31.786195 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.787335 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.788980 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.789713 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.789836 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.789983 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.790145 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.790238 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260619 16:39:31.790392 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.784534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44659 (local address 127.0.0.1:48360)
0619 16:39:31.784873 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.784909 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.785054 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.785892 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.785898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.785922 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:31.786066 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.786073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.787468 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.787471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.788852 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.788863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.789837 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.789840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.789940 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.789975 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.790116 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.790243 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260619 16:39:31.790563 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.785642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39055 (local address 127.0.0.1:58154)
0619 16:39:31.786028 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.786042 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.786130 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.786700 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.786703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.786720 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:31.786848 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.786857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.788050 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.788053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.789102 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.789108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.790168 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.790171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.790271 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.790295 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.790348 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.790412 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":96,"threads_started":1}
I20260619 16:39:31.791431 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.786065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.98.1:39055)
0619 16:39:31.786355 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.786367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.786385 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:39:31.786441 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.786444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.786572 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.786723 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:31.786974 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.787901 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.789208 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.790030 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.791021 (+ 991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.791146 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.791193 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.791281 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260619 16:39:31.793231 17677 tablet_service.cc:1467] Processing CreateTablet for tablet 27787bd3732946439639dad3406fe70b (DEFAULT_TABLE table=test-table [id=51714be2d3174658a167dbd2d1388b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:31.793648 17543 tablet_service.cc:1467] Processing CreateTablet for tablet 27787bd3732946439639dad3406fe70b (DEFAULT_TABLE table=test-table [id=51714be2d3174658a167dbd2d1388b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:31.794234 17677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27787bd3732946439639dad3406fe70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:31.794656 17543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27787bd3732946439639dad3406fe70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:31.797710 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.785481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.98.3:43175)
0619 16:39:31.785872 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.785884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.786081 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:39:31.786302 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.786308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.786479 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.786595 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:31.786990 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.787919 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.789267 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.790030 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.790145 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.790281 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.797437 (+ 7156us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.797554 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":88,"threads_started":1}
I20260619 16:39:31.798002 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.785304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43175 (local address 127.0.0.1:51348)
0619 16:39:31.785986 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.785998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.786147 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.786771 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.786774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.786791 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:31.786875 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.786881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.788069 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.788075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.789135 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.789145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.790340 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.790345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.790420 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.790437 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.790499 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.790550 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":23,"thread_start_us":82,"threads_started":1}
I20260619 16:39:31.800746 17894 tablet_bootstrap.cc:492] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117: Bootstrap starting.
I20260619 16:39:31.800665 17811 tablet_service.cc:1467] Processing CreateTablet for tablet 27787bd3732946439639dad3406fe70b (DEFAULT_TABLE table=test-table [id=51714be2d3174658a167dbd2d1388b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:31.801662 17811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27787bd3732946439639dad3406fe70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:31.801620 17895 tablet_bootstrap.cc:492] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Bootstrap starting.
I20260619 16:39:31.803392 17895 tablet_bootstrap.cc:654] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:31.804378 17895 log.cc:824] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:31.805019 17894 tablet_bootstrap.cc:654] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:31.806262 17895 tablet_bootstrap.cc:492] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: No bootstrap required, opened a new log
I20260619 16:39:31.806414 17894 log.cc:824] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:31.806463 17895 ts_tablet_manager.cc:1397] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:39:31.807849 17898 tablet_bootstrap.cc:492] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Bootstrap starting.
I20260619 16:39:31.808332 17894 tablet_bootstrap.cc:492] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117: No bootstrap required, opened a new log
I20260619 16:39:31.808568 17894 ts_tablet_manager.cc:1397] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 16:39:31.810036 17898 tablet_bootstrap.cc:654] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:31.810268 17895 raft_consensus.cc:348] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.810582 17895 raft_consensus.cc:374] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:31.810680 17895 raft_consensus.cc:729] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e1e023d3f94839b558a1b742b00999, State: Initialized, Role: FOLLOWER
I20260619 16:39:31.811123 17895 consensus_queue.cc:260] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.811246 17898 log.cc:824] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:31.811769 17743 heartbeater.cc:503] Master 127.0.98.62:33917 was elected leader, sending a full tablet report...
I20260619 16:39:31.811741 17894 raft_consensus.cc:348] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.812055 17894 raft_consensus.cc:374] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:31.812096 17895 ts_tablet_manager.cc:1428] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:39:31.812160 17894 raft_consensus.cc:729] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07552443907e4eef9536e60834f5f117, State: Initialized, Role: FOLLOWER
I20260619 16:39:31.812660 17894 consensus_queue.cc:260] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.813581 17894 ts_tablet_manager.cc:1428] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 16:39:31.813670 17609 heartbeater.cc:503] Master 127.0.98.62:33917 was elected leader, sending a full tablet report...
I20260619 16:39:31.815460 17898 tablet_bootstrap.cc:492] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: No bootstrap required, opened a new log
I20260619 16:39:31.815655 17898 ts_tablet_manager.cc:1397] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 16:39:31.818686 17898 raft_consensus.cc:348] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.818890 17898 raft_consensus.cc:374] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:31.818924 17898 raft_consensus.cc:729] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c5ad038fe8464a91aedc02b75624c3, State: Initialized, Role: FOLLOWER
I20260619 16:39:31.819274 17898 consensus_queue.cc:260] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.819859 17877 heartbeater.cc:503] Master 127.0.98.62:33917 was elected leader, sending a full tablet report...
I20260619 16:39:31.820047 17898 ts_tablet_manager.cc:1428] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260619 16:39:31.991683 17878 tablet.cc:2369] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:39:31.994176 17899 raft_consensus.cc:482] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:31.994378 17899 raft_consensus.cc:504] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:31.995579 17899 leader_election.cc:288] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c5ad038fe8464a91aedc02b75624c3 (127.0.98.3:43175), 07552443907e4eef9536e60834f5f117 (127.0.98.1:39055)
I20260619 16:39:32.000025 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.995794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43175 (local address 127.0.98.2:36079)
0619 16:39:31.995964 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.995981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.996069 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.996370 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.996374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.996393 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:31.996542 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.996550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.997459 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.997464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.998482 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.998492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.999565 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.999571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.999686 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.999717 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 16:39:31.999790 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:39:31.999869 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 16:39:32.000101 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.995992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36079 (local address 127.0.98.3:43175)
0619 16:39:31.996102 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.996113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.996130 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:39:31.996184 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.996186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.996251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.996347 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:31.996667 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.997290 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.998615 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.999432 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.999569 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.999661 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.999824 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:39:31.999935 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 16:39:32.000159 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.996370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59219 (local address 127.0.98.1:39055)
0619 16:39:31.996506 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:31.996513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:31.996529 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:39:31.996588 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:31.996593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:31.996647 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:31.996749 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:31.997019 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.997931 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.999114 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.999516 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.999614 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.999678 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:31.999970 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.000042 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:39:32.000437 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:31.996068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39055 (local address 127.0.98.2:59219)
0619 16:39:31.996403 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:31.996416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:31.996485 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:31.996768 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:31.996771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:31.996789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:31.996898 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.996904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.998057 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.998060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.998996 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:31.999003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:31.999741 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:31.999747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:31.999845 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:31.999863 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.000282 (+ 419us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.000340 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":10,"thread_start_us":83,"threads_started":1}
I20260619 16:39:32.001286 17563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27787bd3732946439639dad3406fe70b" candidate_uuid: "91e1e023d3f94839b558a1b742b00999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07552443907e4eef9536e60834f5f117" is_pre_election: true
I20260619 16:39:32.001302 17831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27787bd3732946439639dad3406fe70b" candidate_uuid: "91e1e023d3f94839b558a1b742b00999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c5ad038fe8464a91aedc02b75624c3" is_pre_election: true
I20260619 16:39:32.001652 17831 raft_consensus.cc:2454] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e1e023d3f94839b558a1b742b00999 in term 0.
I20260619 16:39:32.001688 17563 raft_consensus.cc:2454] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91e1e023d3f94839b558a1b742b00999 in term 0.
I20260619 16:39:32.002130 17631 leader_election.cc:302] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c5ad038fe8464a91aedc02b75624c3, 91e1e023d3f94839b558a1b742b00999; no voters:
I20260619 16:39:32.002398 17899 raft_consensus.cc:2781] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:39:32.002481 17899 raft_consensus.cc:482] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:39:32.002532 17899 raft_consensus.cc:3037] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:32.003526 17899 raft_consensus.cc:504] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:32.003882 17899 leader_election.cc:288] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [CANDIDATE]: Term 1 election: Requested vote from peers 15c5ad038fe8464a91aedc02b75624c3 (127.0.98.3:43175), 07552443907e4eef9536e60834f5f117 (127.0.98.1:39055)
I20260619 16:39:32.004218 17831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27787bd3732946439639dad3406fe70b" candidate_uuid: "91e1e023d3f94839b558a1b742b00999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c5ad038fe8464a91aedc02b75624c3"
I20260619 16:39:32.004338 17831 raft_consensus.cc:3037] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:32.004354 17563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27787bd3732946439639dad3406fe70b" candidate_uuid: "91e1e023d3f94839b558a1b742b00999" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07552443907e4eef9536e60834f5f117"
I20260619 16:39:32.004459 17563 raft_consensus.cc:3037] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:32.005434 17831 raft_consensus.cc:2454] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e1e023d3f94839b558a1b742b00999 in term 1.
I20260619 16:39:32.005693 17563 raft_consensus.cc:2454] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91e1e023d3f94839b558a1b742b00999 in term 1.
I20260619 16:39:32.005725 17631 leader_election.cc:302] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c5ad038fe8464a91aedc02b75624c3, 91e1e023d3f94839b558a1b742b00999; no voters:
I20260619 16:39:32.005884 17899 raft_consensus.cc:2781] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:32.006078 17899 raft_consensus.cc:686] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 1 LEADER]: Becoming Leader. State: Replica: 91e1e023d3f94839b558a1b742b00999, State: Running, Role: LEADER
I20260619 16:39:32.006369 17899 consensus_queue.cc:237] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } }
I20260619 16:39:32.009240 17416 catalog_manager.cc:5495] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91e1e023d3f94839b558a1b742b00999 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "91e1e023d3f94839b558a1b742b00999" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e1e023d3f94839b558a1b742b00999" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 } health_report { overall_health: UNKNOWN } } }
I20260619 16:39:32.023129 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.019322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48370 (local address 127.0.98.2:44659)
0619 16:39:32.019486 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.019490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.019679 (+ 189us) server_negotiation.cc:407] Connection header received
0619 16:39:32.019860 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.019863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.019923 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.020026 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:39:32.020278 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.020782 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.021739 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.021967 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.022075 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.022377 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:39:32.022754 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:39:32.022877 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.022919 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.022969 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 16:39:32.026602 17744 tablet.cc:2369] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:39:32.029035 17563 raft_consensus.cc:1261] T 27787bd3732946439639dad3406fe70b P 07552443907e4eef9536e60834f5f117 [term 1 FOLLOWER]: Refusing update from remote peer 91e1e023d3f94839b558a1b742b00999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:32.029117 17831 raft_consensus.cc:1261] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 1 FOLLOWER]: Refusing update from remote peer 91e1e023d3f94839b558a1b742b00999: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:32.029808 17899 consensus_queue.cc:1035] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07552443907e4eef9536e60834f5f117" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:32.029985 17904 consensus_queue.cc:1035] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c5ad038fe8464a91aedc02b75624c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:32.041882 17909 mvcc.cc:205] Tried to move back new op lower bound from 7298609856622616576 to 7298609856539340800. Current Snapshot: MvccSnapshot[applied={T|T < 7298609856622616576}]
I20260619 16:39:32.047667 17418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:32.047753 17908 mvcc.cc:205] Tried to move back new op lower bound from 7298609856622616576 to 7298609856539340800. Current Snapshot: MvccSnapshot[applied={T|T < 7298609856622616576}]
I20260619 16:39:32.047842 17418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:32.050638 17418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e50da62dc31346dfbd3a5b5d47bd857c: Sending DeleteTablet for 3 replicas of tablet 27787bd3732946439639dad3406fe70b
I20260619 16:39:32.051522 17543 tablet_service.cc:1514] Processing DeleteTablet for tablet 27787bd3732946439639dad3406fe70b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58154
I20260619 16:39:32.051632 17677 tablet_service.cc:1514] Processing DeleteTablet for tablet 27787bd3732946439639dad3406fe70b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48360
I20260619 16:39:32.052019 17811 tablet_service.cc:1514] Processing DeleteTablet for tablet 27787bd3732946439639dad3406fe70b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51348
I20260619 16:39:32.052592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17480
I20260619 16:39:32.052598 17917 tablet_replica.cc:330] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: stopping tablet replica
I20260619 16:39:32.052829 17918 tablet_replica.cc:330] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: stopping tablet replica
I20260619 16:39:32.053174 17918 raft_consensus.cc:2229] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:32.053512 17917 raft_consensus.cc:2229] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:39:32.053874 17918 raft_consensus.cc:2258] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:32.054278 17917 raft_consensus.cc:2258] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:32.055912 17918 ts_tablet_manager.cc:1905] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:32.058845 17918 ts_tablet_manager.cc:1918] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:32.058940 17918 log.cc:1196] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/27787bd3732946439639dad3406fe70b
I20260619 16:39:32.059263 17918 ts_tablet_manager.cc:1939] T 27787bd3732946439639dad3406fe70b P 15c5ad038fe8464a91aedc02b75624c3: Deleting consensus metadata
I20260619 16:39:32.060480 17403 catalog_manager.cc:4841] TS 15c5ad038fe8464a91aedc02b75624c3 (127.0.98.3:43175): tablet 27787bd3732946439639dad3406fe70b (table test-table [id=51714be2d3174658a167dbd2d1388b6f]) successfully deleted
I20260619 16:39:32.061141 17917 ts_tablet_manager.cc:1905] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:32.063656 17917 ts_tablet_manager.cc:1918] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:32.063745 17917 log.cc:1196] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/27787bd3732946439639dad3406fe70b
I20260619 16:39:32.064029 17917 ts_tablet_manager.cc:1939] T 27787bd3732946439639dad3406fe70b P 91e1e023d3f94839b558a1b742b00999: Deleting consensus metadata
I20260619 16:39:32.065053 17403 catalog_manager.cc:4841] TS 91e1e023d3f94839b558a1b742b00999 (127.0.98.2:44659): tablet 27787bd3732946439639dad3406fe70b (table test-table [id=51714be2d3174658a167dbd2d1388b6f]) successfully deleted
W20260619 16:39:32.068105 17403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39055 (error 108)
I20260619 16:39:32.068998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17614
I20260619 16:39:32.069953 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.068917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39055 (local address 127.0.0.1:58168)
0619 16:39:32.069775 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.069852 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689}
W20260619 16:39:32.070694 17403 catalog_manager.cc:4584] TS 07552443907e4eef9536e60834f5f117 (127.0.98.1:39055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27787bd3732946439639dad3406fe70b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39055: connect: Connection refused (error 111)
I20260619 16:39:32.075316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17748
I20260619 16:39:32.080257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17386
2026-06-19T16:39:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2436 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T16:39:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:32Z Disabled control of system clock
I20260619 16:39:32.108553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45417
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:39:32.219235 17923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:32.219484 17923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:32.219520 17923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:32.223446 17923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:32.223496 17923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:32.223511 17923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:32.223528 17923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:32.223543 17923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:32.228137 17923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45417
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:32.229069 17923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:32.230355 17923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:32.238456 17931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.238468 17929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.238473 17928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:32.238860 17923 server_base.cc:1034] running on GCE node
I20260619 16:39:32.239353 17923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:32.240402 17923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:32.241626 17923 hybrid_clock.cc:648] HybridClock initialized: now 1781887172241590 us; error 46 us; skew 500 ppm
I20260619 16:39:32.243454 17923 webserver.cc:460] Webserver started at http://127.0.98.62:46819/ using document root <none> and password file <none>
I20260619 16:39:32.244022 17923 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:32.244127 17923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:32.244364 17923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:32.246403 17923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "7c49b64ee9a1418a9079501df243bf1b"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "5228428896b30e7c17702ebe7ed8e8da"
server_key_iv: "bc25d08d7caaba5d5506311ac9715a8a"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.246973 17923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c49b64ee9a1418a9079501df243bf1b"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "5228428896b30e7c17702ebe7ed8e8da"
server_key_iv: "bc25d08d7caaba5d5506311ac9715a8a"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.250556 17923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:32.252928 17937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.254132 17923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:32.254268 17923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "7c49b64ee9a1418a9079501df243bf1b"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "5228428896b30e7c17702ebe7ed8e8da"
server_key_iv: "bc25d08d7caaba5d5506311ac9715a8a"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.254377 17923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:32.279848 17923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:32.280558 17923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:32.280777 17923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:32.287634 17923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45417
I20260619 16:39:32.287642 17989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45417 every 8 connection(s)
I20260619 16:39:32.288548 17923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:32.291685 17990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:32.294461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 17923
I20260619 16:39:32.294648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:32.294953 392 external_mini_cluster.cc:1428] Setting key 780268a2bc9924563d5a049454f2c2f0
I20260619 16:39:32.297549 17990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b: Bootstrap starting.
I20260619 16:39:32.299799 17990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:32.300513 17990 log.cc:824] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:32.302356 17990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b: No bootstrap required, opened a new log
I20260619 16:39:32.304107 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.296383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.62:45417)
0619 16:39:32.297000 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.297016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.297052 (+ 36us) server_negotiation.cc:407] Connection header received
0619 16:39:32.297899 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.297918 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.298301 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.298568 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:32.298906 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.299882 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.300832 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.301375 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.301545 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.302388 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:32.302393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:32.302406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:32.302835 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:32.302889 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:32.302910 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:32.303022 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.303149 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.303395 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260619 16:39:32.306290 17990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c49b64ee9a1418a9079501df243bf1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260619 16:39:32.306572 17990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:32.306622 17990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c49b64ee9a1418a9079501df243bf1b, State: Initialized, Role: FOLLOWER
I20260619 16:39:32.306972 17990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c49b64ee9a1418a9079501df243bf1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260619 16:39:32.307080 17990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:32.307118 17990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:32.307185 17990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:32.307965 17990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c49b64ee9a1418a9079501df243bf1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260619 16:39:32.308236 17990 leader_election.cc:302] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c49b64ee9a1418a9079501df243bf1b; no voters:
I20260619 16:39:32.308466 17990 leader_election.cc:288] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:32.308637 17995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:32.308955 17995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [term 1 LEADER]: Becoming Leader. State: Replica: 7c49b64ee9a1418a9079501df243bf1b, State: Running, Role: LEADER
I20260619 16:39:32.309262 17995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c49b64ee9a1418a9079501df243bf1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } }
I20260619 16:39:32.309659 17990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:32.310861 17996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c49b64ee9a1418a9079501df243bf1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c49b64ee9a1418a9079501df243bf1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } } }
I20260619 16:39:32.310985 17996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:32.311319 17997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c49b64ee9a1418a9079501df243bf1b. Latest consensus state: current_term: 1 leader_uuid: "7c49b64ee9a1418a9079501df243bf1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c49b64ee9a1418a9079501df243bf1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45417 } } }
I20260619 16:39:32.311419 17997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:32.311385 18004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:32.314275 18004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:32.319048 18004 catalog_manager.cc:1347] Generated new cluster ID: 22e97ccfd33c4d7f879f406849d5636e
I20260619 16:39:32.319115 18004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:32.332496 18004 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:32.333737 18004 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:32.343883 18004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b: Generated new TSK 0
I20260619 16:39:32.344555 18004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:32.348371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--builtin_ntp_servers=127.0.98.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:39:32.470134 18014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:32.470422 18014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:32.470489 18014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:32.474584 18014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:32.474665 18014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:32.474761 18014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:32.479475 18014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:32.480579 18014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:32.481990 18014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:32.491616 18022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.491626 18019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.491881 18020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:32.492038 18014 server_base.cc:1034] running on GCE node
I20260619 16:39:32.492429 18014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:32.493706 18014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:32.494879 18014 hybrid_clock.cc:648] HybridClock initialized: now 1781887172494849 us; error 43 us; skew 500 ppm
I20260619 16:39:32.496927 18014 webserver.cc:460] Webserver started at http://127.0.98.1:33269/ using document root <none> and password file <none>
I20260619 16:39:32.497604 18014 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:32.497670 18014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:32.497870 18014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:32.499814 18014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "f59b5a52e8994ebca4b397004cfd3ceb"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "a91a260496a8f949e575c4aa7579531d"
server_key_iv: "9bde32e32fd63f5e2885dc72b58c16f0"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.500389 18014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f59b5a52e8994ebca4b397004cfd3ceb"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "a91a260496a8f949e575c4aa7579531d"
server_key_iv: "9bde32e32fd63f5e2885dc72b58c16f0"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.503970 18014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 16:39:32.506381 18028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.507568 18014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:32.507691 18014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "f59b5a52e8994ebca4b397004cfd3ceb"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "a91a260496a8f949e575c4aa7579531d"
server_key_iv: "9bde32e32fd63f5e2885dc72b58c16f0"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.507802 18014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:32.519101 18014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:32.519677 18014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:32.519856 18014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:32.520375 18014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:32.521286 18014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:32.521361 18014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.521420 18014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:32.521458 18014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.530735 18014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45415
I20260619 16:39:32.530757 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45415 every 8 connection(s)
I20260619 16:39:32.531591 18014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:32.535040 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18014
I20260619 16:39:32.535178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:39:32.535506 392 external_mini_cluster.cc:1428] Setting key 83300c2ebc82d363cf5fee805f537937
I20260619 16:39:32.537678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--builtin_ntp_servers=127.0.98.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:39:32.541615 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.533397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50721 (local address 127.0.98.62:45417)
0619 16:39:32.533551 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.533556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.534597 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 16:39:32.536403 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.536406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.536452 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.536529 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:32.537616 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.538233 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.539643 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.539892 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.540006 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.541104 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:32.541108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:32.541111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:32.541124 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:32.541150 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:32.541158 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:32.541236 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.541422 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.541479 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:39:32.542389 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.533868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45417 (local address 127.0.98.1:50721)
0619 16:39:32.534444 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.534483 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:32.536168 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:32.536668 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:32.536674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:32.536753 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:32.537469 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.537491 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.538361 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.538368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.539505 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.539515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.540040 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.540044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.540205 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.540823 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:32.540825 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:32.540862 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:32.540943 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:32.541238 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:32.541250 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:32.541257 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.541409 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.541679 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":101,"threads_started":1}
I20260619 16:39:32.543723 18142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45417
I20260619 16:39:32.543939 18142 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:32.544476 18142 heartbeater.cc:511] Master 127.0.98.62:45417 requested a full tablet report, sending...
I20260619 16:39:32.546110 17954 ts_manager.cc:194] Registered new tserver with Master: f59b5a52e8994ebca4b397004cfd3ceb (127.0.98.1:45415)
I20260619 16:39:32.547822 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50721
W20260619 16:39:32.646953 18145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:32.647195 18145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:32.647233 18145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:32.651046 18145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:32.651086 18145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:32.651141 18145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:32.655977 18145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:32.656912 18145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:32.658268 18145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:39:32.667972 18145 server_base.cc:1034] running on GCE node
W20260619 16:39:32.668010 18150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.668119 18151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.668015 18153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:32.668524 18145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:32.669188 18145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:32.670367 18145 hybrid_clock.cc:648] HybridClock initialized: now 1781887172670335 us; error 44 us; skew 500 ppm
I20260619 16:39:32.672430 18145 webserver.cc:460] Webserver started at http://127.0.98.2:39149/ using document root <none> and password file <none>
I20260619 16:39:32.673027 18145 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:32.673122 18145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:32.673415 18145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:32.675441 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "79ace53dd7a9d6da037ef484e280a1f7"
server_key_iv: "716bf1cc962a26be3232b3985d6ab169"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.676002 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "79ace53dd7a9d6da037ef484e280a1f7"
server_key_iv: "716bf1cc962a26be3232b3985d6ab169"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.679605 18145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:32.681988 18159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.683203 18145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 16:39:32.683354 18145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "79ace53dd7a9d6da037ef484e280a1f7"
server_key_iv: "716bf1cc962a26be3232b3985d6ab169"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.683467 18145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:32.707405 18145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:32.708091 18145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:32.708294 18145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:32.708864 18145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:32.709918 18145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:32.709990 18145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.710047 18145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:32.710072 18145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.719606 18145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33091
I20260619 16:39:32.719635 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33091 every 8 connection(s)
I20260619 16:39:32.720463 18145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:39:32.723999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18145
I20260619 16:39:32.724125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:39:32.724475 392 external_mini_cluster.cc:1428] Setting key 5386cf17fd83fcf02954deaec8aa8bdd
I20260619 16:39:32.726794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--builtin_ntp_servers=127.0.98.20:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:39:32.731038 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.722314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49887 (local address 127.0.98.62:45417)
0619 16:39:32.722458 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.722464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.723511 (+ 1047us) server_negotiation.cc:407] Connection header received
0619 16:39:32.725457 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.725459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.725502 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.725569 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:32.726685 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.727296 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.728818 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.729221 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.729311 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.730529 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:32.730533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:32.730536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:32.730555 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:32.730574 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:32.730582 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:32.730672 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.730831 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.730895 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 16:39:32.731892 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.722757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45417 (local address 127.0.98.2:49887)
0619 16:39:32.723370 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.723409 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:32.725193 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:32.725763 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:32.725772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:32.725879 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:32.726506 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.726530 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.727430 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.727436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.728672 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.728681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.729450 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.729453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.729600 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.730245 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:32.730246 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:32.730295 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:32.730358 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:32.730673 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:32.730685 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:32.730693 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.730832 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.731127 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260619 16:39:32.733142 18273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45417
I20260619 16:39:32.733409 18273 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:32.733940 18273 heartbeater.cc:511] Master 127.0.98.62:45417 requested a full tablet report, sending...
I20260619 16:39:32.735016 17954 ts_manager.cc:194] Registered new tserver with Master: ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b (127.0.98.2:33091)
I20260619 16:39:32.735919 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49887
W20260619 16:39:32.839783 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:32.840011 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:32.840047 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:32.844199 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:32.844290 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:32.844352 18276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:32.849370 18276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:32.850353 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:32.851577 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:32.861048 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:32.861032 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:32.861258 18276 server_base.cc:1034] running on GCE node
W20260619 16:39:32.861042 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:32.864764 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:32.865523 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:32.868835 18276 hybrid_clock.cc:648] HybridClock initialized: now 1781887172868813 us; error 34 us; skew 500 ppm
I20260619 16:39:32.870715 18276 webserver.cc:460] Webserver started at http://127.0.98.3:40227/ using document root <none> and password file <none>
I20260619 16:39:32.871280 18276 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:32.871373 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:32.871608 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:32.873653 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "4e620133c4f345389cdd86b98b091d62"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "c132b78e4ed8bf11d31f247a8ae0027f"
server_key_iv: "9cb5561051b98efce89a0554de5628b7"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.874202 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4e620133c4f345389cdd86b98b091d62"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "c132b78e4ed8bf11d31f247a8ae0027f"
server_key_iv: "9cb5561051b98efce89a0554de5628b7"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.877810 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:32.880167 18290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.881443 18276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:32.881568 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "4e620133c4f345389cdd86b98b091d62"
format_stamp: "Formatted at 2026-06-19 16:39:32 on dist-test-slave-0f28"
server_key: "c132b78e4ed8bf11d31f247a8ae0027f"
server_key_iv: "9cb5561051b98efce89a0554de5628b7"
server_key_version: "encryptionkey@0"
I20260619 16:39:32.881702 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:32.907536 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:32.908216 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:32.908414 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:32.908991 18276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:32.909930 18276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:32.910001 18276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.910126 18276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:32.910176 18276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:32.919253 18276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39623
I20260619 16:39:32.919284 18403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39623 every 8 connection(s)
I20260619 16:39:32.920099 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:32.923458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18276
I20260619 16:39:32.923605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:39:32.923875 392 external_mini_cluster.cc:1428] Setting key eb189da464f2953bf9350e50a0ca2855
I20260619 16:39:32.929297 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.921785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58871 (local address 127.0.98.62:45417)
0619 16:39:32.921926 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.921930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.922956 (+ 1026us) server_negotiation.cc:407] Connection header received
0619 16:39:32.924453 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.924456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.924509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.924608 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:32.925760 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.926418 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.927510 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.927759 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.927847 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.928887 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:32.928890 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:32.928892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:32.928903 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:32.928920 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:32.928928 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:32.928996 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.929132 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.929181 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 16:39:32.930114 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.922211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45417 (local address 127.0.98.3:58871)
0619 16:39:32.922831 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.922874 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:32.924241 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:32.924818 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:32.924829 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:32.924932 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:32.925614 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.925631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.926528 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.926531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.927391 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.927400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.927885 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.927887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.928019 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.928642 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:32.928643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:32.928683 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:32.928742 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:32.929009 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:32.929015 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:32.929019 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.929109 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.929374 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260619 16:39:32.931130 18404 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45417
I20260619 16:39:32.931309 18404 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:32.931756 18404 heartbeater.cc:511] Master 127.0.98.62:45417 requested a full tablet report, sending...
I20260619 16:39:32.932698 17954 ts_manager.cc:194] Registered new tserver with Master: 4e620133c4f345389cdd86b98b091d62 (127.0.98.3:39623)
I20260619 16:39:32.933197 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58871
I20260619 16:39:32.937631 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:39:32.943166 17993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.939524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.98.62:45417)
0619 16:39:32.939648 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.939652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.939868 (+ 216us) server_negotiation.cc:407] Connection header received
0619 16:39:32.939913 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.939915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.939950 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.940022 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:32.940287 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.940811 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.941644 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.941851 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.941934 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.942778 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:32.942781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:32.942783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:32.942793 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:32.942810 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:32.942817 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:32.942880 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.943019 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.943065 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:39:32.945798 17954 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:39:32.948691 17954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:39:32.962515 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.956694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33091 (local address 127.0.0.1:54842)
0619 16:39:32.957488 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.957536 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:32.957658 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:32.958224 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:32.958229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:32.958247 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:32.958398 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.958405 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.959751 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.959758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.961184 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.961193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.961981 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.961985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.962099 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.962137 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.962250 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.962363 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":89,"threads_started":1}
I20260619 16:39:32.962522 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.956814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.0.98.2:33091)
0619 16:39:32.957728 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.957740 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.957765 (+ 25us) server_negotiation.cc:407] Connection header received
0619 16:39:32.957839 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.957843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.958042 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.958564 (+ 522us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:32.958729 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.959637 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.961306 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.961847 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.961971 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.962155 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.962286 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.962374 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":120,"threads_started":1}
I20260619 16:39:32.963649 18418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.956733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39623 (local address 127.0.0.1:34636)
0619 16:39:32.958075 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.958091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:32.958237 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:32.958882 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:32.958885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:32.958901 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:32.958982 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.958987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.960441 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.960444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.961883 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.961894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.963268 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.963272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.963355 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.963375 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.963444 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.963509 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":77,"thread_start_us":121,"threads_started":1}
I20260619 16:39:32.963878 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.957161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.0.98.3:39623)
0619 16:39:32.957621 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.957634 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.958172 (+ 538us) server_negotiation.cc:407] Connection header received
0619 16:39:32.958415 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.958421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.958608 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.958722 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:32.959146 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.960309 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.962021 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.962796 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.962880 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.963037 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.963569 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.963713 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":170,"threads_started":1}
I20260619 16:39:32.965797 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.957221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45415 (local address 127.0.0.1:41740)
0619 16:39:32.958126 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:32.958140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:32.958235 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:32.958851 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:32.958853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:32.958867 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:32.958935 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.958939 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.960441 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.960444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.961883 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:32.961894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.963104 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.963124 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:32.963238 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.963263 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:39:32.965555 (+ 2292us) client_negotiation.cc:240] Negotiation successful
0619 16:39:32.965629 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":134,"thread_start_us":109,"threads_started":1}
I20260619 16:39:32.965273 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3f48df1f534fdd949ed75ccdd37bfa (DEFAULT_TABLE table=test-table [id=220c577a9ed04547bef00e21e959d697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:32.965896 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:32.957664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.98.1:45415)
0619 16:39:32.957975 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:32.957987 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:32.958345 (+ 358us) server_negotiation.cc:407] Connection header received
0619 16:39:32.958422 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:32.958432 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:32.958571 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:32.958731 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:32.959044 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.960309 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.962021 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:32.962796 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:32.962893 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:32.963037 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:32.965586 (+ 2549us) server_negotiation.cc:299] Negotiation successful
0619 16:39:32.965734 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260619 16:39:32.966470 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3f48df1f534fdd949ed75ccdd37bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:32.968039 18338 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3f48df1f534fdd949ed75ccdd37bfa (DEFAULT_TABLE table=test-table [id=220c577a9ed04547bef00e21e959d697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:32.968259 18076 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3f48df1f534fdd949ed75ccdd37bfa (DEFAULT_TABLE table=test-table [id=220c577a9ed04547bef00e21e959d697]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:32.969040 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3f48df1f534fdd949ed75ccdd37bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:32.969062 18076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3f48df1f534fdd949ed75ccdd37bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:32.972450 18421 tablet_bootstrap.cc:492] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Bootstrap starting.
I20260619 16:39:32.974400 18421 tablet_bootstrap.cc:654] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:32.975106 18421 log.cc:824] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:32.976519 18423 tablet_bootstrap.cc:492] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Bootstrap starting.
I20260619 16:39:32.976729 18424 tablet_bootstrap.cc:492] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: Bootstrap starting.
I20260619 16:39:32.977011 18421 tablet_bootstrap.cc:492] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: No bootstrap required, opened a new log
I20260619 16:39:32.977195 18421 ts_tablet_manager.cc:1397] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:39:32.978413 18424 tablet_bootstrap.cc:654] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:32.978621 18423 tablet_bootstrap.cc:654] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:32.979126 18424 log.cc:824] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:32.979382 18423 log.cc:824] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:32.981082 18424 tablet_bootstrap.cc:492] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: No bootstrap required, opened a new log
I20260619 16:39:32.981040 18421 raft_consensus.cc:348] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:32.981302 18421 raft_consensus.cc:374] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:32.981303 18424 ts_tablet_manager.cc:1397] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:39:32.981387 18421 raft_consensus.cc:729] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b, State: Initialized, Role: FOLLOWER
I20260619 16:39:32.981825 18421 consensus_queue.cc:260] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:32.982795 18421 ts_tablet_manager.cc:1428] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:39:32.983160 18273 heartbeater.cc:503] Master 127.0.98.62:45417 was elected leader, sending a full tablet report...
I20260619 16:39:32.983307 18423 tablet_bootstrap.cc:492] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: No bootstrap required, opened a new log
I20260619 16:39:32.983491 18423 ts_tablet_manager.cc:1397] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260619 16:39:32.984707 18424 raft_consensus.cc:348] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:32.984939 18424 raft_consensus.cc:374] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:32.984997 18424 raft_consensus.cc:729] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59b5a52e8994ebca4b397004cfd3ceb, State: Initialized, Role: FOLLOWER
I20260619 16:39:32.985464 18424 consensus_queue.cc:260] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:32.986127 18142 heartbeater.cc:503] Master 127.0.98.62:45417 was elected leader, sending a full tablet report...
I20260619 16:39:32.986482 18424 ts_tablet_manager.cc:1428] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:39:32.987682 18423 raft_consensus.cc:348] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:32.987916 18423 raft_consensus.cc:374] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:32.987983 18423 raft_consensus.cc:729] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e620133c4f345389cdd86b98b091d62, State: Initialized, Role: FOLLOWER
I20260619 16:39:32.988366 18423 consensus_queue.cc:260] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:32.988987 18404 heartbeater.cc:503] Master 127.0.98.62:45417 was elected leader, sending a full tablet report...
I20260619 16:39:32.989290 18423 ts_tablet_manager.cc:1428] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 16:39:33.023934 18428 raft_consensus.cc:482] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:33.024108 18428 raft_consensus.cc:504] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:33.025133 18428 leader_election.cc:288] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b (127.0.98.2:33091), 4e620133c4f345389cdd86b98b091d62 (127.0.98.3:39623)
I20260619 16:39:33.029054 18419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:33.025579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53805 (local address 127.0.98.3:39623)
0619 16:39:33.025700 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:33.025705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:33.025775 (+ 70us) server_negotiation.cc:407] Connection header received
0619 16:39:33.025922 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:33.025928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:33.025988 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:33.026064 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:33.026369 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.026946 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.028099 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.028487 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.028565 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:33.028638 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:33.028849 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 16:39:33.028923 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:39:33.029084 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:33.025544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37889 (local address 127.0.98.2:33091)
0619 16:39:33.025743 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:33.025750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:33.025991 (+ 241us) server_negotiation.cc:407] Connection header received
0619 16:39:33.026112 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:33.026115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:33.026173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:33.026251 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:33.026573 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.027123 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.028152 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.028563 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.028666 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:33.028737 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:33.028909 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 16:39:33.028982 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 16:39:33.029064 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:33.025453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39623 (local address 127.0.98.1:53805)
0619 16:39:33.025684 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:33.025702 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:33.025808 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:33.026109 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:33.026112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:33.026130 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:33.026258 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:33.026267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.027076 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.027079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:33.027976 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:33.027985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.028612 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.028618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:33.028720 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:33.028743 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:39:33.028818 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 16:39:33.028903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":25}
I20260619 16:39:33.029083 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:33.025453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33091 (local address 127.0.98.1:37889)
0619 16:39:33.025899 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:33.025913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:33.026006 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:33.026268 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:33.026270 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:33.026284 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:33.026426 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:33.026432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.027232 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.027235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:33.028008 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:33.028017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.028700 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.028704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:33.028770 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:33.028805 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 16:39:33.028882 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 16:39:33.028940 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":62,"thread_start_us":82,"threads_started":1}
I20260619 16:39:33.029791 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3f48df1f534fdd949ed75ccdd37bfa" candidate_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e620133c4f345389cdd86b98b091d62" is_pre_election: true
I20260619 16:39:33.029826 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3f48df1f534fdd949ed75ccdd37bfa" candidate_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" is_pre_election: true
I20260619 16:39:33.030092 18227 raft_consensus.cc:2454] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f59b5a52e8994ebca4b397004cfd3ceb in term 0.
I20260619 16:39:33.030092 18358 raft_consensus.cc:2454] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f59b5a52e8994ebca4b397004cfd3ceb in term 0.
I20260619 16:39:33.030586 18029 leader_election.cc:302] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e620133c4f345389cdd86b98b091d62, ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b, f59b5a52e8994ebca4b397004cfd3ceb; no voters:
I20260619 16:39:33.030831 18428 raft_consensus.cc:2781] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:39:33.030907 18428 raft_consensus.cc:482] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:39:33.030949 18428 raft_consensus.cc:3037] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:33.031947 18428 raft_consensus.cc:504] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:33.032280 18428 leader_election.cc:288] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [CANDIDATE]: Term 1 election: Requested vote from peers ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b (127.0.98.2:33091), 4e620133c4f345389cdd86b98b091d62 (127.0.98.3:39623)
I20260619 16:39:33.032610 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3f48df1f534fdd949ed75ccdd37bfa" candidate_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b"
W20260619 16:39:33.032697 18143 tablet.cc:2369] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:39:33.032697 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3f48df1f534fdd949ed75ccdd37bfa" candidate_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e620133c4f345389cdd86b98b091d62"
I20260619 16:39:33.032763 18227 raft_consensus.cc:3037] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:33.032836 18358 raft_consensus.cc:3037] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:33.033924 18227 raft_consensus.cc:2454] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59b5a52e8994ebca4b397004cfd3ceb in term 1.
I20260619 16:39:33.034065 18358 raft_consensus.cc:2454] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59b5a52e8994ebca4b397004cfd3ceb in term 1.
I20260619 16:39:33.034389 18032 leader_election.cc:302] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e620133c4f345389cdd86b98b091d62, f59b5a52e8994ebca4b397004cfd3ceb; no voters:
I20260619 16:39:33.034588 18428 raft_consensus.cc:2781] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:33.034811 18428 raft_consensus.cc:686] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 1 LEADER]: Becoming Leader. State: Replica: f59b5a52e8994ebca4b397004cfd3ceb, State: Running, Role: LEADER
I20260619 16:39:33.035203 18428 consensus_queue.cc:237] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } }
I20260619 16:39:33.038442 17954 catalog_manager.cc:5495] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb reported cstate change: term changed from 0 to 1, leader changed from <none> to f59b5a52e8994ebca4b397004cfd3ceb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f59b5a52e8994ebca4b397004cfd3ceb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45415 } health_report { overall_health: HEALTHY } } }
I20260619 16:39:33.063126 18420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:33.059592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.98.1:45415)
0619 16:39:33.059737 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:33.059742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:33.059761 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:39:33.059841 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:33.059844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:33.059907 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:33.060019 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:39:33.060297 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.060845 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.061761 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:33.061972 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:33.062067 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:33.062311 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:39:33.062698 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:39:33.062786 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:33.062861 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:39:33.062906 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:39:33.068564 18227 raft_consensus.cc:1261] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 1 FOLLOWER]: Refusing update from remote peer f59b5a52e8994ebca4b397004cfd3ceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:33.068576 18358 raft_consensus.cc:1261] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 1 FOLLOWER]: Refusing update from remote peer f59b5a52e8994ebca4b397004cfd3ceb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:33.069252 18431 consensus_queue.cc:1035] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e620133c4f345389cdd86b98b091d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:33.069492 18428 consensus_queue.cc:1035] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:33.080531 18435 mvcc.cc:205] Tried to move back new op lower bound from 7298609860882440192 to 7298609860755255296. Current Snapshot: MvccSnapshot[applied={T|T < 7298609860882440192 or (T in {7298609860882440192})}]
I20260619 16:39:33.088526 17953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:33.088758 17953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:33.091341 17953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7c49b64ee9a1418a9079501df243bf1b: Sending DeleteTablet for 3 replicas of tablet 6e3f48df1f534fdd949ed75ccdd37bfa
I20260619 16:39:33.092142 18338 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3f48df1f534fdd949ed75ccdd37bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:33 UTC) from {username='slave'} at 127.0.0.1:34636
I20260619 16:39:33.092383 18076 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3f48df1f534fdd949ed75ccdd37bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:33 UTC) from {username='slave'} at 127.0.0.1:41740
I20260619 16:39:33.092670 18444 tablet_replica.cc:330] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: stopping tablet replica
I20260619 16:39:33.092679 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3f48df1f534fdd949ed75ccdd37bfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:33 UTC) from {username='slave'} at 127.0.0.1:54842
I20260619 16:39:33.092829 18445 tablet_replica.cc:330] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb: stopping tablet replica
I20260619 16:39:33.093001 18444 raft_consensus.cc:2229] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:33.093029 18445 raft_consensus.cc:2229] T 6e3f48df1f534fdd949ed75ccdd37bfa P f59b5a52e8994ebca4b397004cfd3ceb [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:39:33.093250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18014
I20260619 16:39:33.093380 18446 tablet_replica.cc:330] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: stopping tablet replica
I20260619 16:39:33.093794 18446 raft_consensus.cc:2229] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:33.094169 18446 raft_consensus.cc:2258] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:33.093348 18444 raft_consensus.cc:2258] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:33.096190 18446 ts_tablet_manager.cc:1905] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:33.096623 18444 ts_tablet_manager.cc:1905] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:33.098997 18444 ts_tablet_manager.cc:1918] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:33.099110 18444 log.cc:1196] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/6e3f48df1f534fdd949ed75ccdd37bfa
I20260619 16:39:33.099386 18444 ts_tablet_manager.cc:1939] T 6e3f48df1f534fdd949ed75ccdd37bfa P 4e620133c4f345389cdd86b98b091d62: Deleting consensus metadata
I20260619 16:39:33.099761 18446 ts_tablet_manager.cc:1918] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:33.099864 18446 log.cc:1196] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/6e3f48df1f534fdd949ed75ccdd37bfa
I20260619 16:39:33.100189 18446 ts_tablet_manager.cc:1939] T 6e3f48df1f534fdd949ed75ccdd37bfa P ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b: Deleting consensus metadata
I20260619 16:39:33.100621 17941 catalog_manager.cc:4841] TS 4e620133c4f345389cdd86b98b091d62 (127.0.98.3:39623): tablet 6e3f48df1f534fdd949ed75ccdd37bfa (table test-table [id=220c577a9ed04547bef00e21e959d697]) successfully deleted
I20260619 16:39:33.101361 17938 catalog_manager.cc:4841] TS ca1c6d9fb50b4f0cb6fbfc3ffb6ac71b (127.0.98.2:33091): tablet 6e3f48df1f534fdd949ed75ccdd37bfa (table test-table [id=220c577a9ed04547bef00e21e959d697]) successfully deleted
W20260619 16:39:33.101428 17939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45415 (error 108)
I20260619 16:39:33.101833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18145
I20260619 16:39:33.103634 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:33.103358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45415 (local address 127.0.0.1:41752)
0619 16:39:33.103489 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:33.103562 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 16:39:33.103885 17939 catalog_manager.cc:4584] TS f59b5a52e8994ebca4b397004cfd3ceb (127.0.98.1:45415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e3f48df1f534fdd949ed75ccdd37bfa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45415: connect: Connection refused (error 111)
I20260619 16:39:33.107693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18276
I20260619 16:39:33.113399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 17923
2026-06-19T16:39:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1033 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:33 dist-test-slave-0f28 krb5kdc[18450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:33 dist-test-slave-0f28 krb5kdc[18450](info): set up 2 sockets
Jun 19 16:39:33 dist-test-slave-0f28 krb5kdc[18450](info): commencing operation
krb5kdc: starting...
W20260619 16:39:34.177737 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:34 dist-test-slave-0f28 krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:39:34.307932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40247
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:34.414208 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:34.414494 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:34.414563 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:34.418584 18463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:34.418663 18463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:34.418700 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:34.418761 18463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:34.418800 18463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:34.423885 18463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:34.424885 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:34.425911 18463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:34.433933 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:34.433933 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:34.434119 18463 server_base.cc:1034] running on GCE node
W20260619 16:39:34.433933 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:34.434749 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:34.435786 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:34.436990 18463 hybrid_clock.cc:648] HybridClock initialized: now 1781887174436958 us; error 43 us; skew 500 ppm
Jun 19 16:39:34 dist-test-slave-0f28 krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:34.445765 18463 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:39:34.446599 18463 webserver.cc:460] Webserver started at http://127.0.98.62:40613/ using document root <none> and password file <none>
I20260619 16:39:34.447108 18463 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:34.447175 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:34.447342 18463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:34.449290 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "a290f7863df24c058273c4e662faf855"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "723a44865051d905fa080ef14d920200"
server_key_iv: "d798b2d3bed8f9f8f555bb255cb8fafc"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.449855 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "a290f7863df24c058273c4e662faf855"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "723a44865051d905fa080ef14d920200"
server_key_iv: "d798b2d3bed8f9f8f555bb255cb8fafc"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.453701 18463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 16:39:34.456151 18478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.457428 18463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:34.457569 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "a290f7863df24c058273c4e662faf855"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "723a44865051d905fa080ef14d920200"
server_key_iv: "d798b2d3bed8f9f8f555bb255cb8fafc"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.457679 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:34.481339 18463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:34.484469 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:34.484757 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:34.491641 18463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40247
I20260619 16:39:34.491662 18530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40247 every 8 connection(s)
I20260619 16:39:34.492973 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:34.493760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18463
I20260619 16:39:34.493898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:34.494223 392 external_mini_cluster.cc:1428] Setting key 58106eac7a7bf32fd02224db67b8282a
I20260619 16:39:34.496165 18531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:34.501437 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855: Bootstrap starting.
Jun 19 16:39:34 dist-test-slave-0f28 krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:34.503782 18531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:34.504472 18531 log.cc:824] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:34.506225 18531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855: No bootstrap required, opened a new log
I20260619 16:39:34.508041 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:34.495584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.0.98.62:40247)
0619 16:39:34.496087 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:34.496102 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:34.496137 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:39:34.496838 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:34.496854 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:34.497174 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:34.497557 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:34.498368 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:34.499339 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:34.500228 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:34.500692 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:34.500861 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:34.503577 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:34.503581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:34.503594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:34.503619 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:34.505289 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:34.505812 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:34.505818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:34.505829 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:34.505935 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:34.506219 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:34.506222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:34.506224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:34.506638 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:34.506835 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:34.507119 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 16:39:34.507399 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":143,"threads_started":1}
I20260619 16:39:34.509639 18531 raft_consensus.cc:348] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290f7863df24c058273c4e662faf855" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40247 } }
I20260619 16:39:34.509872 18531 raft_consensus.cc:374] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:34.509959 18531 raft_consensus.cc:729] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a290f7863df24c058273c4e662faf855, State: Initialized, Role: FOLLOWER
I20260619 16:39:34.510357 18531 consensus_queue.cc:260] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290f7863df24c058273c4e662faf855" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40247 } }
I20260619 16:39:34.510483 18531 raft_consensus.cc:388] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:34.510566 18531 raft_consensus.cc:482] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:34.510700 18531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:34.511518 18531 raft_consensus.cc:504] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290f7863df24c058273c4e662faf855" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40247 } }
I20260619 16:39:34.511848 18531 leader_election.cc:302] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a290f7863df24c058273c4e662faf855; no voters:
I20260619 16:39:34.512142 18531 leader_election.cc:288] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:34.512284 18536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:34.512555 18536 raft_consensus.cc:686] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [term 1 LEADER]: Becoming Leader. State: Replica: a290f7863df24c058273c4e662faf855, State: Running, Role: LEADER
I20260619 16:39:34.512848 18536 consensus_queue.cc:237] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290f7863df24c058273c4e662faf855" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40247 } }
I20260619 16:39:34.513298 18531 sys_catalog.cc:564] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:34.514465 18538 sys_catalog.cc:455] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a290f7863df24c058273c4e662faf855. Latest consensus state: current_term: 1 leader_uuid: "a290f7863df24c058273c4e662faf855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290f7863df24c058273c4e662faf855" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40247 } } }
I20260619 16:39:34.514605 18538 sys_catalog.cc:458] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:34.514918 18537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a290f7863df24c058273c4e662faf855" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a290f7863df24c058273c4e662faf855" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40247 } } }
I20260619 16:39:34.514994 18545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:34.515027 18537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:34.518038 18545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:34.523144 18545 catalog_manager.cc:1347] Generated new cluster ID: 773fb4f3a30046de925736ab7a5a3d60
I20260619 16:39:34.523212 18545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:34.529287 18545 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:34.530074 18545 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:34.536723 18545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855: Generated new TSK 0
I20260619 16:39:34.537456 18545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:39:34.568413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40247
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:34.695185 18557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:34.695406 18557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:34.695446 18557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:34.699342 18557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:34.699393 18557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:34.699445 18557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:34.704327 18557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:34.705238 18557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:34.706199 18557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:34.715068 18562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:34.715103 18565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:34.715121 18563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:34.715399 18557 server_base.cc:1034] running on GCE node
I20260619 16:39:34.715760 18557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:34.716367 18557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:34.717568 18557 hybrid_clock.cc:648] HybridClock initialized: now 1781887174717537 us; error 42 us; skew 500 ppm
Jun 19 16:39:34 dist-test-slave-0f28 krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:34.720427 18557 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:39:34.721218 18557 webserver.cc:460] Webserver started at http://127.0.98.1:39235/ using document root <none> and password file <none>
I20260619 16:39:34.721778 18557 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:34.721869 18557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:34.722087 18557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:34.724447 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "bd0cb98b31db483e91a2c074baf16513"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "e5afdee3da80d65f972020003f340624"
server_key_iv: "5e5a483f83fbcc99762ee711997acfc6"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.725052 18557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bd0cb98b31db483e91a2c074baf16513"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "e5afdee3da80d65f972020003f340624"
server_key_iv: "5e5a483f83fbcc99762ee711997acfc6"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.728541 18557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:39:34.730867 18572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.731941 18557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 16:39:34.732079 18557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "bd0cb98b31db483e91a2c074baf16513"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "e5afdee3da80d65f972020003f340624"
server_key_iv: "5e5a483f83fbcc99762ee711997acfc6"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.732185 18557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:34.758139 18557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:34.761488 18557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:34.761687 18557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:34.762310 18557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:34.763093 18557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:34.763164 18557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.763259 18557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:34.763311 18557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.772384 18557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38565
I20260619 16:39:34.772408 18685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38565 every 8 connection(s)
I20260619 16:39:34.773228 18557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:34.775754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18557
I20260619 16:39:34.775889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:39:34.776175 392 external_mini_cluster.cc:1428] Setting key cf85f4c9f0aafc75bd0a0a2a151e2c0e
Jun 19 16:39:34 dist-test-slave-0f28 krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:39:34.787482 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:34.774958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55025 (local address 127.0.98.62:40247)
0619 16:39:34.775103 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:34.775108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:34.776033 (+ 925us) server_negotiation.cc:407] Connection header received
0619 16:39:34.777453 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:34.777457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:34.777510 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:34.777616 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:34.779141 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:34.780070 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:34.781198 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:34.781507 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:34.781616 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:34.783939 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:34.783942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:34.783945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:34.783970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:34.785398 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:34.785917 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:34.785920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:34.785921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:34.785997 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:34.786333 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:34.786335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:34.786336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:34.786488 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:34.786703 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:34.787161 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 16:39:34.787278 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:39:34.788311 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:34.775388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40247 (local address 127.0.98.1:55025)
0619 16:39:34.775890 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:34.775928 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:34.777213 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:34.777783 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:34.777792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:34.778283 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:34.778947 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:34.778969 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:34.780204 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:34.780206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:34.781061 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:34.781071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:34.781663 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:34.781667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:34.781829 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:34.782376 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:34.782392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:34.783788 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:34.785503 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:34.785509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:34.785521 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:34.785792 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:34.786088 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:34.786091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:34.786093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:34.786244 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:34.786701 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:34.786707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:34.786933 (+ 226us) client_negotiation.cc:769] Sending connection context
0619 16:39:34.787120 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 16:39:34.787408 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":139,"threads_started":1}
I20260619 16:39:34.789463 18686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40247
I20260619 16:39:34.789708 18686 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:34.790169 18686 heartbeater.cc:511] Master 127.0.98.62:40247 requested a full tablet report, sending...
I20260619 16:39:34.791726 18495 ts_manager.cc:194] Registered new tserver with Master: bd0cb98b31db483e91a2c074baf16513 (127.0.98.1:38565)
I20260619 16:39:34.793411 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:39:34.804059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40247
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:34.914106 18691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:34.914382 18691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:34.914425 18691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:34.918530 18691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:34.918581 18691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:34.918637 18691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:34.923749 18691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:34.924644 18691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:34.925626 18691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:34.934795 18697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:34.934800 18699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:34.934893 18696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:34.935024 18691 server_base.cc:1034] running on GCE node
I20260619 16:39:34.935555 18691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:34.936215 18691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:34.937386 18691 hybrid_clock.cc:648] HybridClock initialized: now 1781887174937379 us; error 31 us; skew 500 ppm
Jun 19 16:39:34 dist-test-slave-0f28 krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:34.940088 18691 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:39:34.940912 18691 webserver.cc:460] Webserver started at http://127.0.98.2:34385/ using document root <none> and password file <none>
I20260619 16:39:34.941560 18691 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:34.941648 18691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:34.941828 18691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:34.943693 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "46924432aacb4b4b9c0ef1cf649d1bf1"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "a70190b5b24cbf9d8859432bb62e5e3a"
server_key_iv: "20a474bf3efac5356d37b18b460d2696"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.944181 18691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46924432aacb4b4b9c0ef1cf649d1bf1"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "a70190b5b24cbf9d8859432bb62e5e3a"
server_key_iv: "20a474bf3efac5356d37b18b460d2696"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.947613 18691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 16:39:34.949903 18706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.951231 18691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:34.951333 18691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "46924432aacb4b4b9c0ef1cf649d1bf1"
format_stamp: "Formatted at 2026-06-19 16:39:34 on dist-test-slave-0f28"
server_key: "a70190b5b24cbf9d8859432bb62e5e3a"
server_key_iv: "20a474bf3efac5356d37b18b460d2696"
server_key_version: "encryptionkey@0"
I20260619 16:39:34.951414 18691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:34.982582 18691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:34.985661 18691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:34.985821 18691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:34.986362 18691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:34.987221 18691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:34.987268 18691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.987308 18691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:34.987323 18691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:34.996529 18691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36621
I20260619 16:39:34.996564 18819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36621 every 8 connection(s)
I20260619 16:39:34.997440 18691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:39:35.000033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18691
I20260619 16:39:35.000185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:39:35.000514 392 external_mini_cluster.cc:1428] Setting key 8d2bba9f986695b7a27369019c047410
Jun 19 16:39:35 dist-test-slave-0f28 krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 16:39:35.011852 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:34.999202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52757 (local address 127.0.98.62:40247)
0619 16:39:34.999339 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:34.999343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.000167 (+ 824us) server_negotiation.cc:407] Connection header received
0619 16:39:35.001835 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.001839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.001889 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.001972 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:35.003124 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.003644 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.004767 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.004992 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.005114 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.008056 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:35.008059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:35.008062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:35.008085 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:35.009692 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:35.010215 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:35.010218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:35.010219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:35.010291 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:35.010611 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:35.010613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:35.010615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:35.010857 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:35.011022 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.011526 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.011645 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 16:39:35.012843 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:34.999497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40247 (local address 127.0.98.2:52757)
0619 16:39:34.999994 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.000055 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.001629 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.002158 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.002165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.002509 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:35.002971 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.002985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.003772 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.003776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.004640 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.004650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.005150 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.005155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.005390 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.005998 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:35.006014 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:35.007875 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:35.009807 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:35.009816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:35.009834 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:35.010120 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:35.010371 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:35.010373 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:35.010375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:35.010501 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:35.011035 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:35.011040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:35.011301 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.011462 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.011809 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260619 16:39:35.014340 18820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40247
I20260619 16:39:35.014591 18820 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:35.015221 18820 heartbeater.cc:511] Master 127.0.98.62:40247 requested a full tablet report, sending...
I20260619 16:39:35.016446 18495 ts_manager.cc:194] Registered new tserver with Master: 46924432aacb4b4b9c0ef1cf649d1bf1 (127.0.98.2:36621)
I20260619 16:39:35.017356 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:39:35.028515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40247
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:35.140125 18825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:35.140355 18825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:35.140398 18825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:35.144357 18825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:35.144407 18825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:35.144462 18825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:35.149421 18825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:35.150298 18825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:35.151192 18825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:35.160472 18830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:35.160540 18825 server_base.cc:1034] running on GCE node
W20260619 16:39:35.160472 18831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:35.160529 18833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:35.161144 18825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:35.161794 18825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:35.163004 18825 hybrid_clock.cc:648] HybridClock initialized: now 1781887175162994 us; error 27 us; skew 500 ppm
Jun 19 16:39:35 dist-test-slave-0f28 krb5kdc[18450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:35.165831 18825 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:39:35.166745 18825 webserver.cc:460] Webserver started at http://127.0.98.3:39333/ using document root <none> and password file <none>
I20260619 16:39:35.167289 18825 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:35.167347 18825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:35.167500 18825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:35.169456 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "f02a032170f64ca082ea168e5d2fd285"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "b5ea66a9b661b031525d006787d5b15b"
server_key_iv: "99877c8c779ec0010868262c14eb6524"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.169955 18825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f02a032170f64ca082ea168e5d2fd285"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "b5ea66a9b661b031525d006787d5b15b"
server_key_iv: "99877c8c779ec0010868262c14eb6524"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.173413 18825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:39:35.175891 18840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.177194 18825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:35.177299 18825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "f02a032170f64ca082ea168e5d2fd285"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "b5ea66a9b661b031525d006787d5b15b"
server_key_iv: "99877c8c779ec0010868262c14eb6524"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.177479 18825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:35.196363 18825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:35.199200 18825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:35.199404 18825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:35.199970 18825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:35.200752 18825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:35.200798 18825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.200860 18825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:35.200893 18825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.209975 18825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40487
I20260619 16:39:35.210032 18953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40487 every 8 connection(s)
I20260619 16:39:35.210855 18825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:35.214684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 18825
I20260619 16:39:35.214846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:39:35.215209 392 external_mini_cluster.cc:1428] Setting key 9fc04c839c4b9a1b78772a4dadff9b71
Jun 19 16:39:35 dist-test-slave-0f28 krb5kdc[18450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:35.224318 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.212753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50717 (local address 127.0.98.62:40247)
0619 16:39:35.212939 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.212945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.213803 (+ 858us) server_negotiation.cc:407] Connection header received
0619 16:39:35.215183 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.215186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.215248 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.215328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:35.216908 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.217500 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.218622 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.218846 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.218958 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.221314 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:35.221357 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:35.221364 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:35.221392 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:35.222574 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:35.223049 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:35.223052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:35.223053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:35.223133 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:35.223439 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:35.223441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:35.223443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:35.223610 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:35.223687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.224110 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.224207 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 16:39:35.225044 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.213055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40247 (local address 127.0.98.3:50717)
0619 16:39:35.213608 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.213674 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.214968 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.215546 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.215558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.216072 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:35.216763 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.216779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.217623 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.217626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.218486 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.218495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.218992 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.218996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.219173 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.219747 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:35.219762 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:35.221169 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:35.222673 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:35.222677 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:35.222687 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:35.222923 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:35.223229 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:35.223231 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:35.223232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:35.223356 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:35.223696 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:35.223701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:35.223901 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.224053 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.224273 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":107,"threads_started":1}
I20260619 16:39:35.226135 18954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40247
I20260619 16:39:35.226326 18954 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:35.226724 18954 heartbeater.cc:511] Master 127.0.98.62:40247 requested a full tablet report, sending...
I20260619 16:39:35.227691 18495 ts_manager.cc:194] Registered new tserver with Master: f02a032170f64ca082ea168e5d2fd285 (127.0.98.3:40487)
I20260619 16:39:35.228317 18495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50717
I20260619 16:39:35.228775 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:39:35.238389 18534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.230796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.0.98.62:40247)
0619 16:39:35.230923 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.230928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.231079 (+ 151us) server_negotiation.cc:407] Connection header received
0619 16:39:35.231148 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.231150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.231185 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.231247 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:35.231917 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.232382 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.233180 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.233446 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.233526 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.235267 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:35.235269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:35.235271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:35.235292 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:35.236644 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:35.237200 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:35.237204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:35.237206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:35.237284 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:35.237622 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:35.237624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:35.237625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:35.237801 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:35.237915 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.238134 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.238228 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:39:35.241086 18495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:39:35.243999 18495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:39:35.257756 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.251958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.0.98.3:40487)
0619 16:39:35.252395 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.252409 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.252839 (+ 430us) server_negotiation.cc:407] Connection header received
0619 16:39:35.253058 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.253064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.253266 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.253420 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:35.253928 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.254781 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.256216 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.257039 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.257145 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.257282 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.257504 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.257598 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":141,"threads_started":1}
I20260619 16:39:35.257761 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.251856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40487 (local address 127.0.0.1:42688)
0619 16:39:35.252704 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.252749 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.252893 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.253634 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.253639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.253659 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:35.253799 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.253806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.254916 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.254918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.256102 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.256110 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.257168 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.257172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.257284 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.257345 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.257462 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.257622 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":79,"threads_started":1}
I20260619 16:39:35.258358 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.251944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38565 (local address 127.0.0.1:37692)
0619 16:39:35.252760 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.252776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.252886 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.253815 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.253818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.253835 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:35.253922 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.253928 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.255609 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.255612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.256796 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.256808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.257984 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.257987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.258089 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.258111 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.258166 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.258230 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":32,"thread_start_us":91,"threads_started":1}
I20260619 16:39:35.259058 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.252334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.0.98.1:38565)
0619 16:39:35.253423 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.253440 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.253464 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:39:35.253542 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.253545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.253682 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.253837 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:35.254014 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.255466 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.256935 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.257844 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.258622 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.258752 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.258802 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.258889 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":107,"threads_started":1}
I20260619 16:39:35.259476 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.252545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36621 (local address 127.0.0.1:54850)
0619 16:39:35.253262 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.253276 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.253403 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.253993 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.253996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.254012 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:35.254130 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.254136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.255525 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.255531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.257949 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.257959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.259108 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.259112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.259205 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.259226 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.259297 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.259358 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":42,"thread_start_us":86,"threads_started":1}
I20260619 16:39:35.259922 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.252842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.0.98.2:36621)
0619 16:39:35.253173 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.253184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.253381 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:39:35.253513 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.253518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.253692 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.253831 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:35.254228 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.255366 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.258094 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.258793 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.258915 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.259035 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.259696 (+ 661us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.259788 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":88,"threads_started":1}
I20260619 16:39:35.261178 18888 tablet_service.cc:1467] Processing CreateTablet for tablet 2538cda763d04def8b0dd9740f9a7483 (DEFAULT_TABLE table=test-table [id=26f5bc20fcd84bcbaea442b2d111b255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:35.261807 18620 tablet_service.cc:1467] Processing CreateTablet for tablet 2538cda763d04def8b0dd9740f9a7483 (DEFAULT_TABLE table=test-table [id=26f5bc20fcd84bcbaea442b2d111b255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:35.262303 18888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2538cda763d04def8b0dd9740f9a7483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:35.262723 18620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2538cda763d04def8b0dd9740f9a7483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:35.262769 18754 tablet_service.cc:1467] Processing CreateTablet for tablet 2538cda763d04def8b0dd9740f9a7483 (DEFAULT_TABLE table=test-table [id=26f5bc20fcd84bcbaea442b2d111b255]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:35.263690 18754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2538cda763d04def8b0dd9740f9a7483. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:35.269131 18971 tablet_bootstrap.cc:492] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513: Bootstrap starting.
I20260619 16:39:35.269969 18973 tablet_bootstrap.cc:492] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Bootstrap starting.
I20260619 16:39:35.270015 18972 tablet_bootstrap.cc:492] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Bootstrap starting.
I20260619 16:39:35.271688 18972 tablet_bootstrap.cc:654] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:35.271811 18971 tablet_bootstrap.cc:654] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:35.272243 18973 tablet_bootstrap.cc:654] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:35.272776 18972 log.cc:824] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:35.273048 18973 log.cc:824] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:35.273038 18971 log.cc:824] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:35.274873 18972 tablet_bootstrap.cc:492] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: No bootstrap required, opened a new log
I20260619 16:39:35.274879 18971 tablet_bootstrap.cc:492] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513: No bootstrap required, opened a new log
I20260619 16:39:35.274879 18973 tablet_bootstrap.cc:492] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: No bootstrap required, opened a new log
I20260619 16:39:35.275074 18972 ts_tablet_manager.cc:1397] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:39:35.275068 18973 ts_tablet_manager.cc:1397] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:39:35.275079 18971 ts_tablet_manager.cc:1397] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 16:39:35.277796 18972 raft_consensus.cc:348] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.278039 18972 raft_consensus.cc:374] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:35.278091 18972 raft_consensus.cc:729] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02a032170f64ca082ea168e5d2fd285, State: Initialized, Role: FOLLOWER
I20260619 16:39:35.278509 18972 consensus_queue.cc:260] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.278805 18971 raft_consensus.cc:348] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.278805 18973 raft_consensus.cc:348] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.279040 18973 raft_consensus.cc:374] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:35.279040 18971 raft_consensus.cc:374] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:35.279091 18973 raft_consensus.cc:729] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46924432aacb4b4b9c0ef1cf649d1bf1, State: Initialized, Role: FOLLOWER
I20260619 16:39:35.279100 18954 heartbeater.cc:503] Master 127.0.98.62:40247 was elected leader, sending a full tablet report...
I20260619 16:39:35.279109 18971 raft_consensus.cc:729] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0cb98b31db483e91a2c074baf16513, State: Initialized, Role: FOLLOWER
I20260619 16:39:35.279449 18972 ts_tablet_manager.cc:1428] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 16:39:35.279476 18971 consensus_queue.cc:260] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.279476 18973 consensus_queue.cc:260] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.280292 18820 heartbeater.cc:503] Master 127.0.98.62:40247 was elected leader, sending a full tablet report...
I20260619 16:39:35.280292 18686 heartbeater.cc:503] Master 127.0.98.62:40247 was elected leader, sending a full tablet report...
I20260619 16:39:35.280366 18971 ts_tablet_manager.cc:1428] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:39:35.280366 18973 ts_tablet_manager.cc:1428] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260619 16:39:35.410844 18979 raft_consensus.cc:482] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:35.411113 18979 raft_consensus.cc:504] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.412220 18979 leader_election.cc:288] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f02a032170f64ca082ea168e5d2fd285 (127.0.98.3:40487), 46924432aacb4b4b9c0ef1cf649d1bf1 (127.0.98.2:36621)
I20260619 16:39:35.417840 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.413000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36323 (local address 127.0.98.3:40487)
0619 16:39:35.413146 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.413151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.413378 (+ 227us) server_negotiation.cc:407] Connection header received
0619 16:39:35.413450 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.413452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.413497 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.413579 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:35.414269 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.414969 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.416393 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.417051 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.417153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.417215 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.417633 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.417712 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 16:39:35.417840 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.412910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51919 (local address 127.0.98.2:36621)
0619 16:39:35.413159 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.413165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.413186 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:39:35.413439 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.413444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.413495 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.413534 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:35.413968 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.414709 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.416396 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.417036 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.417154 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.417215 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.417633 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.417713 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 16:39:35.417884 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.412490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36621 (local address 127.0.98.1:51919)
0619 16:39:35.412983 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.412999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.413229 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.413696 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.413699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.413710 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:35.413869 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.413873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.414824 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.414826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.416221 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.416232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.417203 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.417207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.417371 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.417464 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.417551 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.417707 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":122,"thread_start_us":147,"threads_started":1}
I20260619 16:39:35.417887 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.412490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40487 (local address 127.0.98.1:36323)
0619 16:39:35.413214 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.413226 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.413313 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.413635 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.413640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.413661 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:35.413852 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.413859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.415062 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.415065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.416260 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.416270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.417203 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.417207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.417376 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.417464 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.417551 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.417707 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":332,"thread_start_us":84,"threads_started":1}
I20260619 16:39:35.418541 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2538cda763d04def8b0dd9740f9a7483" candidate_uuid: "bd0cb98b31db483e91a2c074baf16513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" is_pre_election: true
I20260619 16:39:35.418834 18774 raft_consensus.cc:2454] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0cb98b31db483e91a2c074baf16513 in term 0.
I20260619 16:39:35.418823 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2538cda763d04def8b0dd9740f9a7483" candidate_uuid: "bd0cb98b31db483e91a2c074baf16513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02a032170f64ca082ea168e5d2fd285" is_pre_election: true
I20260619 16:39:35.419090 18908 raft_consensus.cc:2454] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd0cb98b31db483e91a2c074baf16513 in term 0.
I20260619 16:39:35.419359 18574 leader_election.cc:302] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46924432aacb4b4b9c0ef1cf649d1bf1, bd0cb98b31db483e91a2c074baf16513; no voters:
I20260619 16:39:35.419632 18979 raft_consensus.cc:2781] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:39:35.419716 18979 raft_consensus.cc:482] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:39:35.419759 18979 raft_consensus.cc:3037] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:35.420786 18979 raft_consensus.cc:504] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.421114 18979 leader_election.cc:288] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [CANDIDATE]: Term 1 election: Requested vote from peers f02a032170f64ca082ea168e5d2fd285 (127.0.98.3:40487), 46924432aacb4b4b9c0ef1cf649d1bf1 (127.0.98.2:36621)
I20260619 16:39:35.421484 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2538cda763d04def8b0dd9740f9a7483" candidate_uuid: "bd0cb98b31db483e91a2c074baf16513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1"
I20260619 16:39:35.421505 18908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2538cda763d04def8b0dd9740f9a7483" candidate_uuid: "bd0cb98b31db483e91a2c074baf16513" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f02a032170f64ca082ea168e5d2fd285"
I20260619 16:39:35.421613 18908 raft_consensus.cc:3037] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:35.421613 18774 raft_consensus.cc:3037] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:35.422838 18908 raft_consensus.cc:2454] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0cb98b31db483e91a2c074baf16513 in term 1.
I20260619 16:39:35.422838 18774 raft_consensus.cc:2454] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd0cb98b31db483e91a2c074baf16513 in term 1.
I20260619 16:39:35.423178 18573 leader_election.cc:302] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd0cb98b31db483e91a2c074baf16513, f02a032170f64ca082ea168e5d2fd285; no voters:
I20260619 16:39:35.423393 18979 raft_consensus.cc:2781] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:35.423600 18979 raft_consensus.cc:686] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [term 1 LEADER]: Becoming Leader. State: Replica: bd0cb98b31db483e91a2c074baf16513, State: Running, Role: LEADER
I20260619 16:39:35.423904 18979 consensus_queue.cc:237] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } }
I20260619 16:39:35.427121 18494 catalog_manager.cc:5495] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd0cb98b31db483e91a2c074baf16513 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "bd0cb98b31db483e91a2c074baf16513" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd0cb98b31db483e91a2c074baf16513" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 } health_report { overall_health: UNKNOWN } } }
I20260619 16:39:35.447018 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.443619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.0.98.1:38565)
0619 16:39:35.443771 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.443775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.443793 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:39:35.443872 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.443874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.443923 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.444013 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:39:35.444251 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.444748 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.445676 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.445881 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.445962 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.446240 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:39:35.446612 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:39:35.446755 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.446888 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.446919 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 16:39:35.452339 18908 raft_consensus.cc:1261] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 1 FOLLOWER]: Refusing update from remote peer bd0cb98b31db483e91a2c074baf16513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:35.452339 18774 raft_consensus.cc:1261] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 1 FOLLOWER]: Refusing update from remote peer bd0cb98b31db483e91a2c074baf16513: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:35.453037 18982 consensus_queue.cc:1035] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46924432aacb4b4b9c0ef1cf649d1bf1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:35.453184 18979 consensus_queue.cc:1035] T 2538cda763d04def8b0dd9740f9a7483 P bd0cb98b31db483e91a2c074baf16513 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02a032170f64ca082ea168e5d2fd285" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:35.459052 18985 mvcc.cc:205] Tried to move back new op lower bound from 7298609870646231040 to 7298609870538715136. Current Snapshot: MvccSnapshot[applied={T|T < 7298609870646231040}]
I20260619 16:39:35.462291 18986 mvcc.cc:205] Tried to move back new op lower bound from 7298609870646231040 to 7298609870538715136. Current Snapshot: MvccSnapshot[applied={T|T < 7298609870646231040}]
I20260619 16:39:35.469023 18491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:35.469190 18491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:35.471626 18491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a290f7863df24c058273c4e662faf855: Sending DeleteTablet for 3 replicas of tablet 2538cda763d04def8b0dd9740f9a7483
I20260619 16:39:35.472342 18888 tablet_service.cc:1514] Processing DeleteTablet for tablet 2538cda763d04def8b0dd9740f9a7483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42688
I20260619 16:39:35.472488 18620 tablet_service.cc:1514] Processing DeleteTablet for tablet 2538cda763d04def8b0dd9740f9a7483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37692
I20260619 16:39:35.472491 18754 tablet_service.cc:1514] Processing DeleteTablet for tablet 2538cda763d04def8b0dd9740f9a7483 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54850
I20260619 16:39:35.472829 18994 tablet_replica.cc:330] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: stopping tablet replica
I20260619 16:39:35.473112 18994 raft_consensus.cc:2229] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:35.473290 18996 tablet_replica.cc:330] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: stopping tablet replica
I20260619 16:39:35.473449 18994 raft_consensus.cc:2258] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:35.473646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18557
I20260619 16:39:35.475339 18996 raft_consensus.cc:2229] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:35.475664 18996 raft_consensus.cc:2258] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:35.476332 18994 ts_tablet_manager.cc:1905] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:35.478302 18996 ts_tablet_manager.cc:1905] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:35.479823 18994 ts_tablet_manager.cc:1918] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:35.479929 18994 log.cc:1196] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/2538cda763d04def8b0dd9740f9a7483
I20260619 16:39:35.480288 18994 ts_tablet_manager.cc:1939] T 2538cda763d04def8b0dd9740f9a7483 P f02a032170f64ca082ea168e5d2fd285: Deleting consensus metadata
I20260619 16:39:35.480459 18996 ts_tablet_manager.cc:1918] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:35.480577 18996 log.cc:1196] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/2538cda763d04def8b0dd9740f9a7483
I20260619 16:39:35.480942 18996 ts_tablet_manager.cc:1939] T 2538cda763d04def8b0dd9740f9a7483 P 46924432aacb4b4b9c0ef1cf649d1bf1: Deleting consensus metadata
W20260619 16:39:35.481195 18480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38565 (error 108)
I20260619 16:39:35.481586 18479 catalog_manager.cc:4841] TS f02a032170f64ca082ea168e5d2fd285 (127.0.98.3:40487): tablet 2538cda763d04def8b0dd9740f9a7483 (table test-table [id=26f5bc20fcd84bcbaea442b2d111b255]) successfully deleted
I20260619 16:39:35.482125 18480 catalog_manager.cc:4841] TS 46924432aacb4b4b9c0ef1cf649d1bf1 (127.0.98.2:36621): tablet 2538cda763d04def8b0dd9740f9a7483 (table test-table [id=26f5bc20fcd84bcbaea442b2d111b255]) successfully deleted
I20260619 16:39:35.482167 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.481886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38565 (local address 127.0.0.1:37718)
0619 16:39:35.481999 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.482081 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 16:39:35.482311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18691
W20260619 16:39:35.482404 18480 catalog_manager.cc:4584] TS bd0cb98b31db483e91a2c074baf16513 (127.0.98.1:38565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2538cda763d04def8b0dd9740f9a7483: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38565: connect: Connection refused (error 111)
I20260619 16:39:35.488312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18825
I20260619 16:39:35.493865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 18463
2026-06-19T16:39:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2380 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T16:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:35Z Disabled control of system clock
I20260619 16:39:35.522526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36061
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:39:35.631883 19001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:35.632159 19001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:35.632196 19001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:35.636314 19001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:35.636371 19001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:35.636387 19001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:35.636404 19001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:35.636449 19001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:35.641287 19001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:35.642371 19001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:35.643728 19001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:39:35.652230 19001 server_base.cc:1034] running on GCE node
W20260619 16:39:35.652148 19009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:35.652195 19006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:35.652153 19007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:35.652900 19001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:35.654022 19001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:35.655221 19001 hybrid_clock.cc:648] HybridClock initialized: now 1781887175655191 us; error 44 us; skew 500 ppm
I20260619 16:39:35.656935 19001 webserver.cc:460] Webserver started at http://127.0.98.62:37911/ using document root <none> and password file <none>
I20260619 16:39:35.657585 19001 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:35.657675 19001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:35.657894 19001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:35.659838 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "d7275d7d17fdb89fdfd67e777b60d8e9"
server_key_iv: "f69d1f7c5340ffdfd4c65e19c03cafcc"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.660390 19001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "d7275d7d17fdb89fdfd67e777b60d8e9"
server_key_iv: "f69d1f7c5340ffdfd4c65e19c03cafcc"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.664160 19001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:39:35.666556 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.667773 19001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:35.667928 19001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "d7275d7d17fdb89fdfd67e777b60d8e9"
server_key_iv: "f69d1f7c5340ffdfd4c65e19c03cafcc"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.668040 19001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:35.696175 19001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:35.696852 19001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:35.697050 19001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:35.703706 19001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36061
I20260619 16:39:35.703734 19067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36061 every 8 connection(s)
I20260619 16:39:35.704560 19001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:35.707500 19068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:35.707991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19001
I20260619 16:39:35.708156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:35.708453 392 external_mini_cluster.cc:1428] Setting key fd0d77573dd792b5f5fc545d514af2c3
I20260619 16:39:35.714630 19068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8: Bootstrap starting.
I20260619 16:39:35.717190 19068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:35.718029 19068 log.cc:824] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:35.718474 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.709738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.0.98.62:36061)
0619 16:39:35.710384 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.710404 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.710445 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:39:35.711176 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.711200 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.711673 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.712006 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:35.712436 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.713967 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.714931 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.715579 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.715780 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.716564 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:35.716570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:35.716583 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:35.717033 (+ 450us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:35.717115 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:35.717144 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:35.717245 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.717454 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.717708 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260619 16:39:35.720839 19068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8: No bootstrap required, opened a new log
I20260619 16:39:35.724455 19068 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260619 16:39:35.724720 19068 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:35.724805 19068 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a1ac6d9e7742a69fb2c21b4ae0adf8, State: Initialized, Role: FOLLOWER
I20260619 16:39:35.725224 19068 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260619 16:39:35.725380 19068 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:35.725467 19068 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:35.725561 19068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:35.726497 19068 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260619 16:39:35.726814 19068 leader_election.cc:302] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8a1ac6d9e7742a69fb2c21b4ae0adf8; no voters:
I20260619 16:39:35.727084 19068 leader_election.cc:288] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:35.727227 19073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:35.727437 19073 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [term 1 LEADER]: Becoming Leader. State: Replica: f8a1ac6d9e7742a69fb2c21b4ae0adf8, State: Running, Role: LEADER
I20260619 16:39:35.727766 19073 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } }
I20260619 16:39:35.728229 19068 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:35.729154 19075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a1ac6d9e7742a69fb2c21b4ae0adf8. Latest consensus state: current_term: 1 leader_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } } }
I20260619 16:39:35.729308 19075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:35.729700 19074 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8a1ac6d9e7742a69fb2c21b4ae0adf8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36061 } } }
I20260619 16:39:35.729789 19074 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:35.729889 19082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:35.732327 19082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:35.737396 19082 catalog_manager.cc:1347] Generated new cluster ID: 47f9f802d2bd45f796965f0d04824210
I20260619 16:39:35.737457 19082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:35.757211 19082 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:35.758399 19082 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:35.775725 19082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8: Generated new TSK 0
I20260619 16:39:35.776485 19082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:35.784910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--builtin_ntp_servers=127.0.98.20:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:39:35.895561 19092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:35.895838 19092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:35.895906 19092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:35.899923 19092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:35.900002 19092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:35.900089 19092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:35.904711 19092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:35.905758 19092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:35.907063 19092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:39:35.916803 19092 server_base.cc:1034] running on GCE node
W20260619 16:39:35.916731 19097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:35.916822 19100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:35.916738 19098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:35.917430 19092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:35.918059 19092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:35.919216 19092 hybrid_clock.cc:648] HybridClock initialized: now 1781887175919185 us; error 42 us; skew 500 ppm
I20260619 16:39:35.920965 19092 webserver.cc:460] Webserver started at http://127.0.98.1:43577/ using document root <none> and password file <none>
I20260619 16:39:35.921583 19092 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:35.921651 19092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:35.921852 19092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:35.923812 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "3597d9095fcd40c895835c29a40732b3"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "46cb7e8ade8d4c51e6547d0f886a55bb"
server_key_iv: "e3a3ced2e91032c6573045fb9f857e14"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.924350 19092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3597d9095fcd40c895835c29a40732b3"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "46cb7e8ade8d4c51e6547d0f886a55bb"
server_key_iv: "e3a3ced2e91032c6573045fb9f857e14"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.927855 19092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:39:35.930397 19106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.931703 19092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:35.931828 19092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "3597d9095fcd40c895835c29a40732b3"
format_stamp: "Formatted at 2026-06-19 16:39:35 on dist-test-slave-0f28"
server_key: "46cb7e8ade8d4c51e6547d0f886a55bb"
server_key_iv: "e3a3ced2e91032c6573045fb9f857e14"
server_key_version: "encryptionkey@0"
I20260619 16:39:35.931931 19092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:35.944514 19092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:35.945128 19092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:35.945263 19092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:35.945842 19092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:35.946707 19092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:35.946753 19092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.946813 19092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:35.946844 19092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:35.955767 19092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40735
I20260619 16:39:35.955798 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40735 every 8 connection(s)
I20260619 16:39:35.957041 19092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:35.961232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19092
I20260619 16:39:35.961400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:39:35.961722 392 external_mini_cluster.cc:1428] Setting key 6ce154a0f4a7667bcc7e5725a2407f91
I20260619 16:39:35.963608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--builtin_ntp_servers=127.0.98.20:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:39:35.966310 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.958838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34273 (local address 127.0.98.62:36061)
0619 16:39:35.958961 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:35.958965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:35.959890 (+ 925us) server_negotiation.cc:407] Connection header received
0619 16:39:35.961355 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:35.961358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:35.961419 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:35.961494 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:35.962414 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.962965 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.964242 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.964468 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.964580 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.965709 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:35.965713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:35.965715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:35.965728 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:35.965754 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:35.965783 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:35.965859 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:35.966038 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:39:35.966110 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:39:35.967212 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:35.959144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36061 (local address 127.0.98.1:34273)
0619 16:39:35.959748 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:35.959787 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:35.961092 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:35.961661 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:35.961668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:35.961758 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:35.962271 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.962285 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.963102 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.963108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.964117 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:35.964127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:35.964614 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:35.964618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:35.964787 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:35.965437 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:35.965438 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:35.965474 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:35.965542 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:35.965890 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:35.965898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:35.965903 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:39:35.966015 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:39:35.966225 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260619 16:39:35.968410 19220 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36061
I20260619 16:39:35.968659 19220 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:35.969295 19220 heartbeater.cc:511] Master 127.0.98.62:36061 requested a full tablet report, sending...
I20260619 16:39:35.970954 19032 ts_manager.cc:194] Registered new tserver with Master: 3597d9095fcd40c895835c29a40732b3 (127.0.98.1:40735)
I20260619 16:39:35.972078 19032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34273
W20260619 16:39:36.071588 19223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:36.071868 19223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:36.071957 19223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:36.075945 19223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:36.075990 19223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:36.076041 19223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:36.080592 19223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:36.081511 19223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:36.082728 19223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:36.092749 19231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:36.092725 19229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:36.092852 19223 server_base.cc:1034] running on GCE node
W20260619 16:39:36.093366 19228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:36.093863 19223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:36.094466 19223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:36.095659 19223 hybrid_clock.cc:648] HybridClock initialized: now 1781887176095643 us; error 24 us; skew 500 ppm
I20260619 16:39:36.097491 19223 webserver.cc:460] Webserver started at http://127.0.98.2:44579/ using document root <none> and password file <none>
I20260619 16:39:36.098063 19223 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:36.098124 19223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:36.098347 19223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:36.100265 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "472e46272c524ea6b52ce567a8075e6b"
format_stamp: "Formatted at 2026-06-19 16:39:36 on dist-test-slave-0f28"
server_key: "f96e4a02a438e62ed75726f97b7fdc65"
server_key_iv: "b02be05ef33a7aa78570a81ba002c7b8"
server_key_version: "encryptionkey@0"
I20260619 16:39:36.100790 19223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "472e46272c524ea6b52ce567a8075e6b"
format_stamp: "Formatted at 2026-06-19 16:39:36 on dist-test-slave-0f28"
server_key: "f96e4a02a438e62ed75726f97b7fdc65"
server_key_iv: "b02be05ef33a7aa78570a81ba002c7b8"
server_key_version: "encryptionkey@0"
I20260619 16:39:36.104350 19223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 16:39:36.106736 19237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:36.107862 19223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:36.107987 19223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "472e46272c524ea6b52ce567a8075e6b"
format_stamp: "Formatted at 2026-06-19 16:39:36 on dist-test-slave-0f28"
server_key: "f96e4a02a438e62ed75726f97b7fdc65"
server_key_iv: "b02be05ef33a7aa78570a81ba002c7b8"
server_key_version: "encryptionkey@0"
I20260619 16:39:36.108093 19223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:36.129035 19223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:36.129794 19223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:36.130014 19223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:36.130574 19223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:36.131467 19223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:36.131537 19223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:36.131600 19223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:36.131650 19223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:36.140496 19223 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41411
I20260619 16:39:36.140514 19350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41411 every 8 connection(s)
I20260619 16:39:36.141424 19223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:39:36.150115 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.143207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50221 (local address 127.0.98.62:36061)
0619 16:39:36.143339 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.143343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.144258 (+ 915us) server_negotiation.cc:407] Connection header received
0619 16:39:36.145557 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.145560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.145607 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.145680 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:36.146504 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.147018 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.148202 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.148430 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.148510 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.149627 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:36.149631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:36.149633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:36.149645 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:36.149663 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:36.149671 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:36.149732 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.149879 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.149947 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:39:36.150116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19223
I20260619 16:39:36.150239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:39:36.150534 392 external_mini_cluster.cc:1428] Setting key d34460288e12cc04fd7d0cd35155f64f
I20260619 16:39:36.150978 19353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.143513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36061 (local address 127.0.98.2:50221)
0619 16:39:36.144111 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.144149 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.145372 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.145830 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.145836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.145912 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:36.146380 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.146391 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.147169 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.147172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.148083 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.148092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.148574 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.148576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.148743 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.149387 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:36.149389 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:36.149431 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:36.149491 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:36.149743 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:36.149750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:36.149754 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.149847 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.150070 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":108,"threads_started":1}
I20260619 16:39:36.152215 19351 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36061
I20260619 16:39:36.152463 19351 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:36.152669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--builtin_ntp_servers=127.0.98.20:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:39:36.153036 19351 heartbeater.cc:511] Master 127.0.98.62:36061 requested a full tablet report, sending...
I20260619 16:39:36.154104 19031 ts_manager.cc:194] Registered new tserver with Master: 472e46272c524ea6b52ce567a8075e6b (127.0.98.2:41411)
I20260619 16:39:36.154698 19031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50221
W20260619 16:39:36.261927 19354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:36.262197 19354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:36.262261 19354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:36.266288 19354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:36.266359 19354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:36.266438 19354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:36.271234 19354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:36.272290 19354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:36.273706 19354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:36.283360 19360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:36.283367 19359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:36.283367 19362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:36.283915 19354 server_base.cc:1034] running on GCE node
I20260619 16:39:36.284358 19354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:36.285035 19354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:36.286221 19354 hybrid_clock.cc:648] HybridClock initialized: now 1781887176286178 us; error 57 us; skew 500 ppm
I20260619 16:39:36.288015 19354 webserver.cc:460] Webserver started at http://127.0.98.3:34617/ using document root <none> and password file <none>
I20260619 16:39:36.288666 19354 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:36.288750 19354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:36.288964 19354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:36.290984 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "68015044faad4a8793085d45cbfb21cd"
format_stamp: "Formatted at 2026-06-19 16:39:36 on dist-test-slave-0f28"
server_key: "7ff25c81e75a0402c10cae9bc6d56b21"
server_key_iv: "b650f8e036d90d2b271fa5150902e0b6"
server_key_version: "encryptionkey@0"
I20260619 16:39:36.291478 19354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68015044faad4a8793085d45cbfb21cd"
format_stamp: "Formatted at 2026-06-19 16:39:36 on dist-test-slave-0f28"
server_key: "7ff25c81e75a0402c10cae9bc6d56b21"
server_key_iv: "b650f8e036d90d2b271fa5150902e0b6"
server_key_version: "encryptionkey@0"
I20260619 16:39:36.295095 19354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 16:39:36.297410 19368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:36.298605 19354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:36.298718 19354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "68015044faad4a8793085d45cbfb21cd"
format_stamp: "Formatted at 2026-06-19 16:39:36 on dist-test-slave-0f28"
server_key: "7ff25c81e75a0402c10cae9bc6d56b21"
server_key_iv: "b650f8e036d90d2b271fa5150902e0b6"
server_key_version: "encryptionkey@0"
I20260619 16:39:36.298805 19354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:36.312703 19354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:36.313447 19354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:36.313589 19354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:36.314098 19354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:36.314962 19354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:36.315007 19354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:36.315044 19354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:36.315059 19354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:36.324144 19354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45379
I20260619 16:39:36.324183 19481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45379 every 8 connection(s)
I20260619 16:39:36.325001 19354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:36.329178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19354
I20260619 16:39:36.329356 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:39:36.329689 392 external_mini_cluster.cc:1428] Setting key 55d876abcd702e28eb2684b1ecff410b
I20260619 16:39:36.334836 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.326865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34961 (local address 127.0.98.62:36061)
0619 16:39:36.327048 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.327054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.328146 (+ 1092us) server_negotiation.cc:407] Connection header received
0619 16:39:36.329840 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.329843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.329891 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.329964 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:36.330993 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.331744 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.332871 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.333120 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.333231 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.334346 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:36.334349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:36.334352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:36.334365 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:36.334388 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:36.334401 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:36.334475 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.334638 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.334702 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 16:39:36.335691 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.327249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36061 (local address 127.0.98.3:34961)
0619 16:39:36.328001 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.328040 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.329607 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.330130 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.330137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.330217 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:36.330843 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.330870 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.331867 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.331870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.332747 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.332755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.333264 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.333268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.333452 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.334067 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:36.334069 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:36.334107 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:36.334165 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:36.334491 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:36.334500 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:36.334505 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.334638 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.334929 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260619 16:39:36.336699 19482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36061
I20260619 16:39:36.336889 19482 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:36.337309 19482 heartbeater.cc:511] Master 127.0.98.62:36061 requested a full tablet report, sending...
I20260619 16:39:36.338275 19031 ts_manager.cc:194] Registered new tserver with Master: 68015044faad4a8793085d45cbfb21cd (127.0.98.3:45379)
I20260619 16:39:36.338832 19031 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34961
I20260619 16:39:36.343341 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:39:36.348640 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.345128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.0.98.62:36061)
0619 16:39:36.345251 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.345255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.345440 (+ 185us) server_negotiation.cc:407] Connection header received
0619 16:39:36.345539 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.345541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.345574 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.345639 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:36.345854 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.346333 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.347174 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.347374 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.347450 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.348276 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:36.348278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:36.348280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:36.348290 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:36.348305 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:36.348312 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:36.348399 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.348509 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.348538 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 16:39:36.351425 19031 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:39:36.354494 19031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:39:36.368650 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.362473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40735 (local address 127.0.0.1:60976)
0619 16:39:36.362808 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.362859 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.362990 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.363603 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.363609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.363634 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:36.363799 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.363808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.365339 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.365346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.366467 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.366477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.367967 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.367970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.368185 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.368236 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.368377 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.368499 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260619 16:39:36.368711 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.362614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.98.1:40735)
0619 16:39:36.363046 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.363076 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.363102 (+ 26us) server_negotiation.cc:407] Connection header received
0619 16:39:36.363183 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.363187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.363399 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.363558 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:36.363945 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.365212 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.366669 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.367870 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.367971 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.368267 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.368406 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.368503 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":122,"threads_started":1}
I20260619 16:39:36.368834 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.362681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41411 (local address 127.0.0.1:47928)
0619 16:39:36.363218 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.363230 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.363301 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.364023 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.364029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.364048 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:36.364159 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.364166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.365907 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.365911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.367062 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.367073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.368453 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.368457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.368544 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.368569 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.368620 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.368688 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":103,"threads_started":1}
I20260619 16:39:36.369434 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.363236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.2:41411)
0619 16:39:36.363553 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.363577 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.363599 (+ 22us) server_negotiation.cc:407] Connection header received
0619 16:39:36.363672 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.363680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.363863 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.364308 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:36.364444 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.365775 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.367200 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.368316 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.368934 (+ 618us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.369100 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.369147 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.369239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260619 16:39:36.369545 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.363167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45379 (local address 127.0.0.1:36206)
0619 16:39:36.364162 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.364177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.364262 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.364847 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.364850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.364866 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:36.364942 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.364947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.366311 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.366317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.367786 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.367798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.369097 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.369101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.369197 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.369219 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.369304 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.369402 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"thread_start_us":81,"threads_started":1}
I20260619 16:39:36.369935 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.363608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.0.98.3:45379)
0619 16:39:36.364004 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.364015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.364349 (+ 334us) server_negotiation.cc:407] Connection header received
0619 16:39:36.364425 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.364429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.364589 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.364719 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:36.365073 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.366191 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.367965 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.368750 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.368850 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.369032 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.369699 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.369790 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":170,"threads_started":1}
I20260619 16:39:36.372283 19416 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed2c4af0c24465e8ff1a87f61474407 (DEFAULT_TABLE table=test-table [id=08b14d45225e42209807ca8ce4a84103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:36.372267 19154 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed2c4af0c24465e8ff1a87f61474407 (DEFAULT_TABLE table=test-table [id=08b14d45225e42209807ca8ce4a84103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:36.372437 19285 tablet_service.cc:1467] Processing CreateTablet for tablet 2ed2c4af0c24465e8ff1a87f61474407 (DEFAULT_TABLE table=test-table [id=08b14d45225e42209807ca8ce4a84103]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:36.373215 19416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed2c4af0c24465e8ff1a87f61474407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:36.373279 19154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed2c4af0c24465e8ff1a87f61474407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:36.373279 19285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ed2c4af0c24465e8ff1a87f61474407. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:36.379829 19499 tablet_bootstrap.cc:492] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Bootstrap starting.
I20260619 16:39:36.380514 19500 tablet_bootstrap.cc:492] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: Bootstrap starting.
I20260619 16:39:36.380898 19501 tablet_bootstrap.cc:492] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Bootstrap starting.
I20260619 16:39:36.382249 19499 tablet_bootstrap.cc:654] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:36.382534 19500 tablet_bootstrap.cc:654] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:36.382576 19501 tablet_bootstrap.cc:654] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:36.383122 19499 log.cc:824] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:36.383224 19500 log.cc:824] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:36.383203 19501 log.cc:824] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:36.385059 19499 tablet_bootstrap.cc:492] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: No bootstrap required, opened a new log
I20260619 16:39:36.385070 19501 tablet_bootstrap.cc:492] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: No bootstrap required, opened a new log
I20260619 16:39:36.385247 19499 ts_tablet_manager.cc:1397] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 16:39:36.385247 19501 ts_tablet_manager.cc:1397] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 16:39:36.385296 19500 tablet_bootstrap.cc:492] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: No bootstrap required, opened a new log
I20260619 16:39:36.385461 19500 ts_tablet_manager.cc:1397] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 16:39:36.388365 19499 raft_consensus.cc:348] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.388592 19499 raft_consensus.cc:374] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:36.388643 19499 raft_consensus.cc:729] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68015044faad4a8793085d45cbfb21cd, State: Initialized, Role: FOLLOWER
I20260619 16:39:36.389150 19499 consensus_queue.cc:260] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.389477 19505 raft_consensus.cc:482] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:36.389468 19501 raft_consensus.cc:348] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.389468 19500 raft_consensus.cc:348] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.389701 19500 raft_consensus.cc:374] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:36.389701 19501 raft_consensus.cc:374] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:36.389761 19501 raft_consensus.cc:729] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472e46272c524ea6b52ce567a8075e6b, State: Initialized, Role: FOLLOWER
I20260619 16:39:36.389768 19500 raft_consensus.cc:729] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3597d9095fcd40c895835c29a40732b3, State: Initialized, Role: FOLLOWER
I20260619 16:39:36.389631 19505 raft_consensus.cc:504] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.390107 19499 ts_tablet_manager.cc:1428] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 16:39:36.390187 19500 consensus_queue.cc:260] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.390187 19501 consensus_queue.cc:260] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.390429 19482 heartbeater.cc:503] Master 127.0.98.62:36061 was elected leader, sending a full tablet report...
I20260619 16:39:36.391095 19500 ts_tablet_manager.cc:1428] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:39:36.391088 19501 ts_tablet_manager.cc:1428] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 16:39:36.391398 19220 heartbeater.cc:503] Master 127.0.98.62:36061 was elected leader, sending a full tablet report...
W20260619 16:39:36.392256 19352 tablet.cc:2369] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:39:36.394116 19505 leader_election.cc:288] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3597d9095fcd40c895835c29a40732b3 (127.0.98.1:40735), 472e46272c524ea6b52ce567a8075e6b (127.0.98.2:41411)
I20260619 16:39:36.394832 19351 heartbeater.cc:503] Master 127.0.98.62:36061 was elected leader, sending a full tablet report...
I20260619 16:39:36.396577 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.391324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40735 (local address 127.0.98.3:36575)
0619 16:39:36.391827 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.391842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.391921 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.392907 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.392911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.392931 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:36.393042 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.393050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.394126 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.394129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.395204 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.395217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.396133 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.396138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.396246 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.396269 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.396355 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.396420 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425}
I20260619 16:39:36.396975 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.391807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36575 (local address 127.0.98.1:40735)
0619 16:39:36.392245 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.392250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.392264 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:39:36.392308 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.392311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.392352 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.392428 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:36.393180 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.393905 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.395358 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.395915 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.396013 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.396072 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.396802 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.396871 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369}
I20260619 16:39:36.397303 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.393387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40839 (local address 127.0.98.2:41411)
0619 16:39:36.393491 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.393498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.393513 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:39:36.393565 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.393570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.393628 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.393704 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:36.393993 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.394669 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.396022 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.396577 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.396664 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.396718 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.397096 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.397178 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:39:36.397468 19174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed2c4af0c24465e8ff1a87f61474407" candidate_uuid: "68015044faad4a8793085d45cbfb21cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3597d9095fcd40c895835c29a40732b3" is_pre_election: true
I20260619 16:39:36.397673 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.393031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41411 (local address 127.0.98.3:40839)
0619 16:39:36.393310 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.393346 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:36.393429 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:36.393729 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:36.393732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:36.393747 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:36.393849 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.393854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.394796 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.394799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.395885 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:36.395896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.396847 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.396853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:36.396945 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.396969 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 16:39:36.397449 (+ 480us) client_negotiation.cc:240] Negotiation successful
0619 16:39:36.397523 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":128,"threads_started":1}
I20260619 16:39:36.397751 19174 raft_consensus.cc:2454] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68015044faad4a8793085d45cbfb21cd in term 0.
I20260619 16:39:36.398217 19372 leader_election.cc:302] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3597d9095fcd40c895835c29a40732b3, 68015044faad4a8793085d45cbfb21cd; no voters:
I20260619 16:39:36.398464 19506 raft_consensus.cc:2781] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:39:36.398543 19506 raft_consensus.cc:482] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:39:36.398607 19506 raft_consensus.cc:3037] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:36.398615 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed2c4af0c24465e8ff1a87f61474407" candidate_uuid: "68015044faad4a8793085d45cbfb21cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472e46272c524ea6b52ce567a8075e6b" is_pre_election: true
I20260619 16:39:36.398837 19305 raft_consensus.cc:2454] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68015044faad4a8793085d45cbfb21cd in term 0.
I20260619 16:39:36.399585 19506 raft_consensus.cc:504] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.399940 19506 leader_election.cc:288] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [CANDIDATE]: Term 1 election: Requested vote from peers 3597d9095fcd40c895835c29a40732b3 (127.0.98.1:40735), 472e46272c524ea6b52ce567a8075e6b (127.0.98.2:41411)
I20260619 16:39:36.400214 19174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed2c4af0c24465e8ff1a87f61474407" candidate_uuid: "68015044faad4a8793085d45cbfb21cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3597d9095fcd40c895835c29a40732b3"
I20260619 16:39:36.400270 19305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ed2c4af0c24465e8ff1a87f61474407" candidate_uuid: "68015044faad4a8793085d45cbfb21cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "472e46272c524ea6b52ce567a8075e6b"
I20260619 16:39:36.400336 19174 raft_consensus.cc:3037] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:36.400358 19305 raft_consensus.cc:3037] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:36.401540 19305 raft_consensus.cc:2454] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68015044faad4a8793085d45cbfb21cd in term 1.
I20260619 16:39:36.401540 19174 raft_consensus.cc:2454] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68015044faad4a8793085d45cbfb21cd in term 1.
I20260619 16:39:36.401823 19370 leader_election.cc:302] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 472e46272c524ea6b52ce567a8075e6b, 68015044faad4a8793085d45cbfb21cd; no voters:
I20260619 16:39:36.402020 19506 raft_consensus.cc:2781] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:36.402164 19506 raft_consensus.cc:686] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 1 LEADER]: Becoming Leader. State: Replica: 68015044faad4a8793085d45cbfb21cd, State: Running, Role: LEADER
I20260619 16:39:36.402449 19506 consensus_queue.cc:237] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } }
I20260619 16:39:36.405645 19032 catalog_manager.cc:5495] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 68015044faad4a8793085d45cbfb21cd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "68015044faad4a8793085d45cbfb21cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68015044faad4a8793085d45cbfb21cd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 } health_report { overall_health: UNKNOWN } } }
I20260619 16:39:36.419075 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.415451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.0.98.3:45379)
0619 16:39:36.415589 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:36.415594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:36.415608 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:39:36.415812 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:36.415815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:36.415867 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:36.415969 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:39:36.416242 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.416731 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.417652 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:36.417874 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:36.417986 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:36.418284 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:39:36.418653 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:39:36.418807 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:36.418920 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:39:36.418954 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:39:36.424562 19305 raft_consensus.cc:1261] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 1 FOLLOWER]: Refusing update from remote peer 68015044faad4a8793085d45cbfb21cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:36.424554 19174 raft_consensus.cc:1261] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 1 FOLLOWER]: Refusing update from remote peer 68015044faad4a8793085d45cbfb21cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:36.425247 19505 consensus_queue.cc:1035] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3597d9095fcd40c895835c29a40732b3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:36.425457 19506 consensus_queue.cc:1035] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "472e46272c524ea6b52ce567a8075e6b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:36.432111 19512 mvcc.cc:205] Tried to move back new op lower bound from 7298609874628386816 to 7298609874547195904. Current Snapshot: MvccSnapshot[applied={T|T < 7298609874628386816}]
I20260619 16:39:36.432694 19513 mvcc.cc:205] Tried to move back new op lower bound from 7298609874628386816 to 7298609874547195904. Current Snapshot: MvccSnapshot[applied={T|T < 7298609874628386816}]
I20260619 16:39:36.442148 19032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:36.442333 19032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:36.444756 19032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8a1ac6d9e7742a69fb2c21b4ae0adf8: Sending DeleteTablet for 3 replicas of tablet 2ed2c4af0c24465e8ff1a87f61474407
I20260619 16:39:36.445597 19285 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed2c4af0c24465e8ff1a87f61474407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:36 UTC) from {username='slave'} at 127.0.0.1:47928
I20260619 16:39:36.445597 19154 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed2c4af0c24465e8ff1a87f61474407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:36 UTC) from {username='slave'} at 127.0.0.1:60976
I20260619 16:39:36.446115 19523 tablet_replica.cc:330] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3: stopping tablet replica
I20260619 16:39:36.446195 19416 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ed2c4af0c24465e8ff1a87f61474407 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:36 UTC) from {username='slave'} at 127.0.0.1:36206
I20260619 16:39:36.446506 19523 raft_consensus.cc:2229] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:36.446718 19524 tablet_replica.cc:330] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: stopping tablet replica
I20260619 16:39:36.446838 19523 raft_consensus.cc:2258] T 2ed2c4af0c24465e8ff1a87f61474407 P 3597d9095fcd40c895835c29a40732b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:36.447081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19092
I20260619 16:39:36.447080 19524 raft_consensus.cc:2229] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:39:36.448136 19524 raft_consensus.cc:2258] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:36.448293 19522 tablet_replica.cc:330] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: stopping tablet replica
I20260619 16:39:36.448585 19522 raft_consensus.cc:2229] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:36.448872 19522 raft_consensus.cc:2258] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:36.449890 19524 ts_tablet_manager.cc:1905] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:36.450692 19522 ts_tablet_manager.cc:1905] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:36.453115 19524 ts_tablet_manager.cc:1918] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:36.453279 19524 log.cc:1196] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/2ed2c4af0c24465e8ff1a87f61474407
I20260619 16:39:36.453536 19522 ts_tablet_manager.cc:1918] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:36.453639 19522 log.cc:1196] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/2ed2c4af0c24465e8ff1a87f61474407
I20260619 16:39:36.453810 19524 ts_tablet_manager.cc:1939] T 2ed2c4af0c24465e8ff1a87f61474407 P 68015044faad4a8793085d45cbfb21cd: Deleting consensus metadata
I20260619 16:39:36.453969 19522 ts_tablet_manager.cc:1939] T 2ed2c4af0c24465e8ff1a87f61474407 P 472e46272c524ea6b52ce567a8075e6b: Deleting consensus metadata
I20260619 16:39:36.455029 19019 catalog_manager.cc:4841] TS 68015044faad4a8793085d45cbfb21cd (127.0.98.3:45379): tablet 2ed2c4af0c24465e8ff1a87f61474407 (table test-table [id=08b14d45225e42209807ca8ce4a84103]) successfully deleted
I20260619 16:39:36.455147 19017 catalog_manager.cc:4841] TS 472e46272c524ea6b52ce567a8075e6b (127.0.98.2:41411): tablet 2ed2c4af0c24465e8ff1a87f61474407 (table test-table [id=08b14d45225e42209807ca8ce4a84103]) successfully deleted
W20260619 16:39:36.456593 19019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40735 (error 108)
I20260619 16:39:36.456702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19223
I20260619 16:39:36.458093 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:36.457834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40735 (local address 127.0.0.1:60986)
0619 16:39:36.457958 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:36.458019 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 16:39:36.458298 19019 catalog_manager.cc:4584] TS 3597d9095fcd40c895835c29a40732b3 (127.0.98.1:40735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ed2c4af0c24465e8ff1a87f61474407: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40735: connect: Connection refused (error 111)
I20260619 16:39:36.464501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19354
I20260619 16:39:36.470324 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19001
2026-06-19T16:39:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (976 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:36 dist-test-slave-0f28 krb5kdc[19528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:36 dist-test-slave-0f28 krb5kdc[19528](info): set up 2 sockets
Jun 19 16:39:36 dist-test-slave-0f28 krb5kdc[19528](info): commencing operation
krb5kdc: starting...
W20260619 16:39:37.521442 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:37 dist-test-slave-0f28 krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:39:37.645149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41937
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:37.750586 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:37.750855 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:37.750934 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:37.754822 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:37.754902 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:37.754940 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:37.754997 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:37.755038 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:37.760015 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:37.760965 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:37.761937 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:37.769971 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:37.769985 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:37.769985 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:37.770335 19541 server_base.cc:1034] running on GCE node
I20260619 16:39:37.770925 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:37.771986 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:37.773191 19541 hybrid_clock.cc:648] HybridClock initialized: now 1781887177773173 us; error 36 us; skew 500 ppm
Jun 19 16:39:37 dist-test-slave-0f28 krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:37.775799 19541 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:39:37.776592 19541 webserver.cc:460] Webserver started at http://127.0.98.62:42125/ using document root <none> and password file <none>
I20260619 16:39:37.777117 19541 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:37.777209 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:37.777470 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:37.779403 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "22f2f0091f0543c1b91b131278d50f99"
format_stamp: "Formatted at 2026-06-19 16:39:37 on dist-test-slave-0f28"
server_key: "4aa5c6d114c7ebd5965832d62c7e6d26"
server_key_iv: "25f7fd910e75da1574046f6802929968"
server_key_version: "encryptionkey@0"
I20260619 16:39:37.779973 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "22f2f0091f0543c1b91b131278d50f99"
format_stamp: "Formatted at 2026-06-19 16:39:37 on dist-test-slave-0f28"
server_key: "4aa5c6d114c7ebd5965832d62c7e6d26"
server_key_iv: "25f7fd910e75da1574046f6802929968"
server_key_version: "encryptionkey@0"
I20260619 16:39:37.783460 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:37.785764 19556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:37.786924 19541 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:37.787057 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "22f2f0091f0543c1b91b131278d50f99"
format_stamp: "Formatted at 2026-06-19 16:39:37 on dist-test-slave-0f28"
server_key: "4aa5c6d114c7ebd5965832d62c7e6d26"
server_key_iv: "25f7fd910e75da1574046f6802929968"
server_key_version: "encryptionkey@0"
I20260619 16:39:37.787165 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:37.809629 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:37.812392 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:37.812611 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:37.819721 19541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41937
I20260619 16:39:37.819736 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41937 every 8 connection(s)
I20260619 16:39:37.820689 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:37.823311 19609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:37.828725 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99: Bootstrap starting.
I20260619 16:39:37.830592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19541
I20260619 16:39:37.830719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:37.831008 392 external_mini_cluster.cc:1428] Setting key 608fecfb3eedc1ffbc7218fc0654470c
I20260619 16:39:37.831166 19609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:37.832046 19609 log.cc:824] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:37.834178 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99: No bootstrap required, opened a new log
I20260619 16:39:37.838539 19609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2f0091f0543c1b91b131278d50f99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41937 } }
I20260619 16:39:37.838774 19609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:37.838848 19609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f2f0091f0543c1b91b131278d50f99, State: Initialized, Role: FOLLOWER
I20260619 16:39:37.839337 19609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2f0091f0543c1b91b131278d50f99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41937 } }
I20260619 16:39:37.839464 19609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:37.839553 19609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:37.839669 19609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 16:39:37 dist-test-slave-0f28 krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:37.840711 19609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2f0091f0543c1b91b131278d50f99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41937 } }
I20260619 16:39:37.841096 19609 leader_election.cc:302] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f2f0091f0543c1b91b131278d50f99; no voters:
I20260619 16:39:37.841511 19609 leader_election.cc:288] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:37.841667 19614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:37.841876 19614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [term 1 LEADER]: Becoming Leader. State: Replica: 22f2f0091f0543c1b91b131278d50f99, State: Running, Role: LEADER
I20260619 16:39:37.842159 19614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2f0091f0543c1b91b131278d50f99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41937 } }
I20260619 16:39:37.842900 19609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:37.843251 19614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f2f0091f0543c1b91b131278d50f99. Latest consensus state: current_term: 1 leader_uuid: "22f2f0091f0543c1b91b131278d50f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2f0091f0543c1b91b131278d50f99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41937 } } }
I20260619 16:39:37.843369 19614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:37.843604 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f2f0091f0543c1b91b131278d50f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f2f0091f0543c1b91b131278d50f99" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41937 } } }
I20260619 16:39:37.843693 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:37.843940 19618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:37.846984 19618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:37.847246 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:37.832113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.0.98.62:41937)
0619 16:39:37.832554 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:37.832568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:37.832601 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:39:37.833404 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:37.833425 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:37.833808 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:37.834145 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:37.834850 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:37.836391 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:37.837217 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:37.837952 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:37.838123 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:37.840704 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:37.840711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:37.840729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:37.840761 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:37.842910 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:37.843477 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:37.843483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:37.843489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:37.843629 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:37.844026 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:37.844039 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:37.844041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:37.844593 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:37.844839 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:37.845788 (+ 949us) server_negotiation.cc:299] Negotiation successful
0619 16:39:37.846130 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260619 16:39:37.852864 19618 catalog_manager.cc:1347] Generated new cluster ID: 788521d74a0a40f6b3beaa9234c950c9
I20260619 16:39:37.852927 19618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:37.872438 19618 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:37.874090 19618 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:37.882227 19618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99: Generated new TSK 0
I20260619 16:39:37.882931 19618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 16:39:37.912139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41937
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:38.021104 19635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:38.021385 19635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:38.021476 19635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:38.025384 19635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:38.025462 19635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:38.025560 19635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:38.030563 19635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:38.031512 19635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:38.032485 19635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:38.041658 19640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:38.041947 19635 server_base.cc:1034] running on GCE node
W20260619 16:39:38.041710 19641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:38.041652 19643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:38.042416 19635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:38.043058 19635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:38.044237 19635 hybrid_clock.cc:648] HybridClock initialized: now 1781887178044209 us; error 40 us; skew 500 ppm
Jun 19 16:39:38 dist-test-slave-0f28 krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:38.046939 19635 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 16:39:38.047762 19635 webserver.cc:460] Webserver started at http://127.0.98.1:36805/ using document root <none> and password file <none>
I20260619 16:39:38.048314 19635 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:38.048373 19635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:38.048656 19635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:38.050631 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "614963eabebc4855aaa86c0d63cf3ec0"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "f87848f6bac3c73f588d241fb618bcaf"
server_key_iv: "47730f6adea2e9cdc4ed555d6b7eca7a"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.051194 19635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "614963eabebc4855aaa86c0d63cf3ec0"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "f87848f6bac3c73f588d241fb618bcaf"
server_key_iv: "47730f6adea2e9cdc4ed555d6b7eca7a"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.054690 19635 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:38.056913 19650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.058164 19635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:38.058295 19635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "614963eabebc4855aaa86c0d63cf3ec0"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "f87848f6bac3c73f588d241fb618bcaf"
server_key_iv: "47730f6adea2e9cdc4ed555d6b7eca7a"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.058400 19635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:38.075664 19635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:38.078665 19635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:38.078859 19635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:38.079440 19635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:38.080240 19635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:38.080312 19635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.080379 19635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:38.080423 19635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.089464 19635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39313
I20260619 16:39:38.089519 19763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39313 every 8 connection(s)
I20260619 16:39:38.090417 19635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:38.101078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19635
I20260619 16:39:38.101215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:39:38.101516 392 external_mini_cluster.cc:1428] Setting key d25262dc90e9ed1572a70e359c329685
Jun 19 16:39:38 dist-test-slave-0f28 krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:38.110385 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.092135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49517 (local address 127.0.98.62:41937)
0619 16:39:38.092298 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.092302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.093235 (+ 933us) server_negotiation.cc:407] Connection header received
0619 16:39:38.094870 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.094875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.094935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.095037 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:38.100890 (+ 5853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.101730 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.103273 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.103643 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.103763 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.106508 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:38.106514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:38.106521 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:38.106543 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:38.108056 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.108580 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.108584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.108586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.108666 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.109012 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.109015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.109017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.109467 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:38.109641 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.110100 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.110221 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 16:39:38.111284 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.092511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41937 (local address 127.0.98.1:49517)
0619 16:39:38.093083 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.093138 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.094595 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.095243 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.095249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.095634 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:38.096147 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.096170 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.101900 (+ 5730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.101904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.103156 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.103166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.103790 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.103792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.103922 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.104507 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:38.104522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:38.106340 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:38.108188 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:38.108194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:38.108208 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:38.108463 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:38.108760 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:38.108763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:38.108765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:38.108907 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:38.109587 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:38.109594 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:38.109861 (+ 267us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.110057 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.110391 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":121,"threads_started":1}
I20260619 16:39:38.112732 19764 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41937
I20260619 16:39:38.113010 19764 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 16:39:38.113665 19764 heartbeater.cc:511] Master 127.0.98.62:41937 requested a full tablet report, sending...
I20260619 16:39:38.115352 19573 ts_manager.cc:194] Registered new tserver with Master: 614963eabebc4855aaa86c0d63cf3ec0 (127.0.98.1:39313)
I20260619 16:39:38.116577 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 16:39:38.131810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41937
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:38.239753 19769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:38.239967 19769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:38.240007 19769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:38.244063 19769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:38.244115 19769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:38.244169 19769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:38.249109 19769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:38.249997 19769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:38.250859 19769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:38.259907 19775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:38.259907 19774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:38.260097 19777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:38.259989 19769 server_base.cc:1034] running on GCE node
I20260619 16:39:38.260504 19769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:38.261057 19769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:38.262241 19769 hybrid_clock.cc:648] HybridClock initialized: now 1781887178262208 us; error 49 us; skew 500 ppm
Jun 19 16:39:38 dist-test-slave-0f28 krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:38.264941 19769 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 16:39:38.265841 19769 webserver.cc:460] Webserver started at http://127.0.98.2:34875/ using document root <none> and password file <none>
I20260619 16:39:38.266361 19769 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:38.266446 19769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:38.266638 19769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:38.268514 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "8194276a114587a6a98f53cc2da8f260"
server_key_iv: "b0f0c87e13d92826072f49911fab1d05"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.269057 19769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "8194276a114587a6a98f53cc2da8f260"
server_key_iv: "b0f0c87e13d92826072f49911fab1d05"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.272590 19769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 16:39:38.274958 19784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.276116 19769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:38.276240 19769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "8194276a114587a6a98f53cc2da8f260"
server_key_iv: "b0f0c87e13d92826072f49911fab1d05"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.276341 19769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:38.303882 19769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:38.306735 19769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:38.306926 19769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:38.307461 19769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:38.308238 19769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:38.308307 19769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.308372 19769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:38.308419 19769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.317286 19769 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44077
I20260619 16:39:38.317309 19897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44077 every 8 connection(s)
I20260619 16:39:38.318181 19769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 16:39:38 dist-test-slave-0f28 krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:38.328042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19769
I20260619 16:39:38.328179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:39:38.328511 392 external_mini_cluster.cc:1428] Setting key abbe0d403b6fad8c83a579e60782d84a
I20260619 16:39:38.331326 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.319907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42043 (local address 127.0.98.62:41937)
0619 16:39:38.320042 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.320046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.320822 (+ 776us) server_negotiation.cc:407] Connection header received
0619 16:39:38.321773 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.321776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.321847 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.321948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:38.323016 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.323542 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.324634 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.324854 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.324965 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.327467 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:38.327470 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:38.327474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:38.327526 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:38.329226 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.329813 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.329815 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.329816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.329925 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.330275 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.330278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.330280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.330449 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:38.330557 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.331049 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.331164 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 16:39:38.332247 19900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.320260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41937 (local address 127.0.98.2:42043)
0619 16:39:38.320688 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.320726 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.321574 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.322114 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.322120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.322446 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:38.322874 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.322886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.323668 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.323672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.324511 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.324519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.324997 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.325000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.325165 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.325765 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:38.325781 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:38.327299 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:38.329377 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:38.329383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:38.329394 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:38.329716 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:38.330015 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:38.330018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:38.330020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:38.330168 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:38.330561 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:38.330569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:38.330794 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.330986 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.331281 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":104,"threads_started":1}
I20260619 16:39:38.333379 19898 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41937
I20260619 16:39:38.333621 19898 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:38.334184 19898 heartbeater.cc:511] Master 127.0.98.62:41937 requested a full tablet report, sending...
I20260619 16:39:38.335232 19573 ts_manager.cc:194] Registered new tserver with Master: e4e99ddc72944e8aa8ac27c9fdcfb043 (127.0.98.2:44077)
I20260619 16:39:38.335776 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42043
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 16:39:38.356324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41937
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:38.469264 19903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:38.469518 19903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:38.469559 19903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:38.473649 19903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:38.473701 19903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:38.473760 19903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:38.478736 19903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:38.479661 19903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:38.480664 19903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:38.489780 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:38.489851 19908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:38.489840 19911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:38.490007 19903 server_base.cc:1034] running on GCE node
I20260619 16:39:38.490557 19903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:38.491189 19903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:38.492419 19903 hybrid_clock.cc:648] HybridClock initialized: now 1781887178492386 us; error 42 us; skew 500 ppm
Jun 19 16:39:38 dist-test-slave-0f28 krb5kdc[19528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:38.495187 19903 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 16:39:38.495944 19903 webserver.cc:460] Webserver started at http://127.0.98.3:35737/ using document root <none> and password file <none>
I20260619 16:39:38.496424 19903 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:38.496479 19903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:38.496637 19903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:38.498536 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3bddf3aa3984f65bd4c7a758dfbe832"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "f682f0c36fff64be983151d6326d13c3"
server_key_iv: "932ff24405a81690ff7edb2dc58a304c"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.499020 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3bddf3aa3984f65bd4c7a758dfbe832"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "f682f0c36fff64be983151d6326d13c3"
server_key_iv: "932ff24405a81690ff7edb2dc58a304c"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.502440 19903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 16:39:38.504659 19918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.506461 19903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:39:38.506551 19903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "e3bddf3aa3984f65bd4c7a758dfbe832"
format_stamp: "Formatted at 2026-06-19 16:39:38 on dist-test-slave-0f28"
server_key: "f682f0c36fff64be983151d6326d13c3"
server_key_iv: "932ff24405a81690ff7edb2dc58a304c"
server_key_version: "encryptionkey@0"
I20260619 16:39:38.506630 19903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:38.522403 19903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:38.525455 19903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:38.525606 19903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:38.526074 19903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:38.526778 19903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:38.526824 19903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.526861 19903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:38.526876 19903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:38.535331 19903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37849
I20260619 16:39:38.535372 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37849 every 8 connection(s)
I20260619 16:39:38.536121 19903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:38.542102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 19903
I20260619 16:39:38.542238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:39:38.542538 392 external_mini_cluster.cc:1428] Setting key dca8dae945d54e94b21b7bfc184739e9
Jun 19 16:39:38 dist-test-slave-0f28 krb5kdc[19528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:38.548806 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.537930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36739 (local address 127.0.98.62:41937)
0619 16:39:38.538064 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.538068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.538679 (+ 611us) server_negotiation.cc:407] Connection header received
0619 16:39:38.539550 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.539553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.539614 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.539690 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:38.540764 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.541305 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.542554 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.542849 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.542959 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.545508 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:38.545512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:38.545514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:38.545540 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:38.546860 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.547368 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.547372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.547374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.547458 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.547796 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.547800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.547802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.547976 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:38.548086 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.548542 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.548654 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:39:38.549603 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.538064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41937 (local address 127.0.98.3:36739)
0619 16:39:38.538489 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.538527 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.539341 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.539846 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.539853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.540177 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:38.540601 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.540613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.541456 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.541460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.542429 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.542438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.542995 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.542999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.543215 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.543856 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:39:38.543875 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:38.545349 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:38.546997 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:38.547003 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:38.547015 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:38.547262 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:38.547548 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:39:38.547552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:39:38.547554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:39:38.547693 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:39:38.548087 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:38.548094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:38.548317 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.548486 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.548815 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":116,"threads_started":1}
I20260619 16:39:38.550623 20032 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41937
I20260619 16:39:38.550802 20032 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:38.551172 20032 heartbeater.cc:511] Master 127.0.98.62:41937 requested a full tablet report, sending...
I20260619 16:39:38.552032 19573 ts_manager.cc:194] Registered new tserver with Master: e3bddf3aa3984f65bd4c7a758dfbe832 (127.0.98.3:37849)
I20260619 16:39:38.552609 19573 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36739
I20260619 16:39:38.556272 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:39:38.565572 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.558304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.0.98.62:41937)
0619 16:39:38.558464 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.558469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.558645 (+ 176us) server_negotiation.cc:407] Connection header received
0619 16:39:38.558791 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.558794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.558840 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.558922 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:38.559619 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.560119 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.560936 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.561153 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.561258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.562538 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:38.562543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:38.562545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:38.562570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:38.563885 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.564329 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.564334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.564335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.564414 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:38.564693 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:38.564697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:38.564699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:38.564915 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:38.565048 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.565282 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.565421 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:39:38.568450 19573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:39:38.571470 19573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:39:38.586520 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.580161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37849 (local address 127.0.0.1:37254)
0619 16:39:38.580797 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.580810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.580897 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.581536 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.581539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.581551 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:38.581671 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.581677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.582975 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.582977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.584609 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.584618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.585963 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.585966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.586091 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.586135 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.586244 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.586362 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":183,"thread_start_us":81,"threads_started":1}
I20260619 16:39:38.586768 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.579700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44077 (local address 127.0.0.1:41638)
0619 16:39:38.580457 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.580470 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.580557 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.581501 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.581507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.581528 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:38.581657 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.581665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.584145 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.584151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.585038 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.585049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.586360 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.586364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.586460 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.586481 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.586564 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.586641 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260619 16:39:38.586820 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.580277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.0.98.2:44077)
0619 16:39:38.580988 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.581000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.581024 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:39:38.581111 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.581116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.581354 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.581507 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:38.583005 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.584009 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.585218 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.586230 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.586360 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.586495 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.586593 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.586681 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":89,"threads_started":1}
I20260619 16:39:38.587085 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.580333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37254 (local address 127.0.98.3:37849)
0619 16:39:38.580802 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.580815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.580981 (+ 166us) server_negotiation.cc:407] Connection header received
0619 16:39:38.581057 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.581061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.581225 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.581390 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:38.581780 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.582855 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.584737 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.585648 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.585751 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.585900 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.586814 (+ 914us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.586909 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":143,"threads_started":1}
I20260619 16:39:38.587909 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.579492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39313 (local address 127.0.0.1:42864)
0619 16:39:38.580245 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.580294 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.580416 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.581234 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.581241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.581264 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:38.581440 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.581452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.582950 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.582954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.584607 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.584624 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.587512 (+ 2888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.587515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.587614 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.587635 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.587707 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.587779 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":264,"thread_start_us":63,"threads_started":1}
I20260619 16:39:38.588347 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.579689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.0.98.1:39313)
0619 16:39:38.580692 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.580710 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.580734 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:39:38.580810 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.580817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.581025 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.581224 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:38.581586 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.582785 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.585405 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.587237 (+ 1832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.587334 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.587454 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.588102 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.588198 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":109,"threads_started":1}
I20260619 16:39:38.589792 19966 tablet_service.cc:1467] Processing CreateTablet for tablet 817012fd7566410fba9b2032900447b8 (DEFAULT_TABLE table=test-table [id=d59ec658df564fa4990b24daf69a12a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:38.589792 19832 tablet_service.cc:1467] Processing CreateTablet for tablet 817012fd7566410fba9b2032900447b8 (DEFAULT_TABLE table=test-table [id=d59ec658df564fa4990b24daf69a12a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:38.590888 19832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817012fd7566410fba9b2032900447b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:38.590888 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817012fd7566410fba9b2032900447b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:38.591952 19698 tablet_service.cc:1467] Processing CreateTablet for tablet 817012fd7566410fba9b2032900447b8 (DEFAULT_TABLE table=test-table [id=d59ec658df564fa4990b24daf69a12a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:38.593298 19698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817012fd7566410fba9b2032900447b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:38.599009 20051 tablet_bootstrap.cc:492] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: Bootstrap starting.
I20260619 16:39:38.599280 20050 tablet_bootstrap.cc:492] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Bootstrap starting.
I20260619 16:39:38.600847 20051 tablet_bootstrap.cc:654] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:38.601240 20050 tablet_bootstrap.cc:654] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:38.601917 20050 log.cc:824] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:38.601917 20051 log.cc:824] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:38.602597 20049 tablet_bootstrap.cc:492] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Bootstrap starting.
I20260619 16:39:38.603832 20051 tablet_bootstrap.cc:492] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: No bootstrap required, opened a new log
I20260619 16:39:38.603832 20050 tablet_bootstrap.cc:492] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: No bootstrap required, opened a new log
I20260619 16:39:38.604040 20050 ts_tablet_manager.cc:1397] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:39:38.604040 20051 ts_tablet_manager.cc:1397] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:39:38.604254 20049 tablet_bootstrap.cc:654] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:38.605225 20049 log.cc:824] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:38.606763 20049 tablet_bootstrap.cc:492] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: No bootstrap required, opened a new log
I20260619 16:39:38.606953 20049 ts_tablet_manager.cc:1397] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:39:38.607892 20050 raft_consensus.cc:348] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.607892 20051 raft_consensus.cc:348] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.608155 20051 raft_consensus.cc:374] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:38.608155 20050 raft_consensus.cc:374] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:38.608214 20051 raft_consensus.cc:729] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 614963eabebc4855aaa86c0d63cf3ec0, State: Initialized, Role: FOLLOWER
I20260619 16:39:38.608214 20050 raft_consensus.cc:729] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e99ddc72944e8aa8ac27c9fdcfb043, State: Initialized, Role: FOLLOWER
I20260619 16:39:38.608649 20051 consensus_queue.cc:260] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.608647 20050 consensus_queue.cc:260] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.609287 19898 heartbeater.cc:503] Master 127.0.98.62:41937 was elected leader, sending a full tablet report...
I20260619 16:39:38.609581 19764 heartbeater.cc:503] Master 127.0.98.62:41937 was elected leader, sending a full tablet report...
I20260619 16:39:38.609651 20050 ts_tablet_manager.cc:1428] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:39:38.610425 20049 raft_consensus.cc:348] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.610700 20049 raft_consensus.cc:374] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:38.610761 20049 raft_consensus.cc:729] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bddf3aa3984f65bd4c7a758dfbe832, State: Initialized, Role: FOLLOWER
I20260619 16:39:38.611217 20049 consensus_queue.cc:260] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.612123 20049 ts_tablet_manager.cc:1428] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 16:39:38.612102 20032 heartbeater.cc:503] Master 127.0.98.62:41937 was elected leader, sending a full tablet report...
I20260619 16:39:38.612694 20051 ts_tablet_manager.cc:1428] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: Time spent starting tablet: real 0.009s user 0.000s sys 0.005s
W20260619 16:39:38.786985 20033 tablet.cc:2369] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:39:38.819308 19899 tablet.cc:2369] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 16:39:38.841671 19765 tablet.cc:2369] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:39:38.885592 20056 raft_consensus.cc:482] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:38.885787 20056 raft_consensus.cc:504] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.886999 20056 leader_election.cc:288] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 614963eabebc4855aaa86c0d63cf3ec0 (127.0.98.1:39313), e3bddf3aa3984f65bd4c7a758dfbe832 (127.0.98.3:37849)
I20260619 16:39:38.891283 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.887358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45513 (local address 127.0.98.1:39313)
0619 16:39:38.887535 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.887540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.887662 (+ 122us) server_negotiation.cc:407] Connection header received
0619 16:39:38.887836 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.887840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.887892 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.887973 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:38.888350 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.888973 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.890222 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.890650 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.890729 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.890809 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.891083 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.891169 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 16:39:38.891304 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.887227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39313 (local address 127.0.98.2:45513)
0619 16:39:38.887565 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.887585 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.887724 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.887995 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.887998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.888063 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:38.888217 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.888224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.889088 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.889091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.890111 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.890120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.890777 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.890781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.890929 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.890957 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.891016 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.891101 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":43,"spinlock_wait_cycles":28800,"thread_start_us":101,"threads_started":1}
I20260619 16:39:38.891355 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.887752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44883 (local address 127.0.98.3:37849)
0619 16:39:38.887888 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.887891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.888032 (+ 141us) server_negotiation.cc:407] Connection header received
0619 16:39:38.888176 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.888181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.888247 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.888326 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:38.888638 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.889146 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.890309 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.890684 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.890822 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.890889 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.891165 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.891249 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 16:39:38.891527 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.887633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37849 (local address 127.0.98.2:44883)
0619 16:39:38.887944 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.887957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:38.888074 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:38.888351 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:38.888353 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:38.888369 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:38.888482 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.888487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.889247 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.889249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.890195 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:38.890206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.890845 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.890849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:38.890961 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.890977 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 16:39:38.891069 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 16:39:38.891138 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":2,"spinlock_wait_cycles":6784,"thread_start_us":93,"threads_started":1}
I20260619 16:39:38.892339 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817012fd7566410fba9b2032900447b8" candidate_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" is_pre_election: true
I20260619 16:39:38.892407 19718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817012fd7566410fba9b2032900447b8" candidate_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614963eabebc4855aaa86c0d63cf3ec0" is_pre_election: true
I20260619 16:39:38.892642 19986 raft_consensus.cc:2454] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e99ddc72944e8aa8ac27c9fdcfb043 in term 0.
I20260619 16:39:38.892661 19718 raft_consensus.cc:2454] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e99ddc72944e8aa8ac27c9fdcfb043 in term 0.
I20260619 16:39:38.893114 19788 leader_election.cc:302] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3bddf3aa3984f65bd4c7a758dfbe832, e4e99ddc72944e8aa8ac27c9fdcfb043; no voters:
I20260619 16:39:38.893394 20056 raft_consensus.cc:2781] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:39:38.893467 20056 raft_consensus.cc:482] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:39:38.893517 20056 raft_consensus.cc:3037] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:38.894531 20056 raft_consensus.cc:504] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.894848 20056 leader_election.cc:288] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [CANDIDATE]: Term 1 election: Requested vote from peers 614963eabebc4855aaa86c0d63cf3ec0 (127.0.98.1:39313), e3bddf3aa3984f65bd4c7a758dfbe832 (127.0.98.3:37849)
I20260619 16:39:38.895179 19718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817012fd7566410fba9b2032900447b8" candidate_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "614963eabebc4855aaa86c0d63cf3ec0"
I20260619 16:39:38.895229 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817012fd7566410fba9b2032900447b8" candidate_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832"
I20260619 16:39:38.895300 19718 raft_consensus.cc:3037] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:38.895344 19986 raft_consensus.cc:3037] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:38.896298 19986 raft_consensus.cc:2454] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e99ddc72944e8aa8ac27c9fdcfb043 in term 1.
I20260619 16:39:38.896416 19718 raft_consensus.cc:2454] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e99ddc72944e8aa8ac27c9fdcfb043 in term 1.
I20260619 16:39:38.896595 19788 leader_election.cc:302] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3bddf3aa3984f65bd4c7a758dfbe832, e4e99ddc72944e8aa8ac27c9fdcfb043; no voters:
I20260619 16:39:38.896816 20056 raft_consensus.cc:2781] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:38.897054 20056 raft_consensus.cc:686] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 1 LEADER]: Becoming Leader. State: Replica: e4e99ddc72944e8aa8ac27c9fdcfb043, State: Running, Role: LEADER
I20260619 16:39:38.897411 20056 consensus_queue.cc:237] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } }
I20260619 16:39:38.900053 19571 catalog_manager.cc:5495] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4e99ddc72944e8aa8ac27c9fdcfb043 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4e99ddc72944e8aa8ac27c9fdcfb043" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 } health_report { overall_health: UNKNOWN } } }
I20260619 16:39:38.949601 20047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.946320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.98.2:44077)
0619 16:39:38.946437 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:38.946441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:38.946455 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:39:38.946529 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:38.946532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:38.946588 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:38.946682 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:39:38.946917 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.947415 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.948241 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:38.948452 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:38.948544 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:38.948776 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:39:38.949166 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:39:38.949253 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:38.949332 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 16:39:38.949424 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:39:38.955679 19718 raft_consensus.cc:1261] T 817012fd7566410fba9b2032900447b8 P 614963eabebc4855aaa86c0d63cf3ec0 [term 1 FOLLOWER]: Refusing update from remote peer e4e99ddc72944e8aa8ac27c9fdcfb043: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:38.955695 19986 raft_consensus.cc:1261] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 1 FOLLOWER]: Refusing update from remote peer e4e99ddc72944e8aa8ac27c9fdcfb043: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:38.956364 20060 consensus_queue.cc:1035] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bddf3aa3984f65bd4c7a758dfbe832" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:38.956593 20056 consensus_queue.cc:1035] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [LEADER]: Connected to new peer: Peer: permanent_uuid: "614963eabebc4855aaa86c0d63cf3ec0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:38.965054 20064 mvcc.cc:205] Tried to move back new op lower bound from 7298609884995194880 to 7298609884765913088. Current Snapshot: MvccSnapshot[applied={T|T < 7298609884995194880 or (T in {7298609884995194880})}]
I20260619 16:39:38.965431 20065 mvcc.cc:205] Tried to move back new op lower bound from 7298609884995194880 to 7298609884765913088. Current Snapshot: MvccSnapshot[applied={T|T < 7298609884995194880}]
I20260619 16:39:38.971262 19572 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:38.971398 19572 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:38.973709 19572 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22f2f0091f0543c1b91b131278d50f99: Sending DeleteTablet for 3 replicas of tablet 817012fd7566410fba9b2032900447b8
I20260619 16:39:38.974413 19832 tablet_service.cc:1514] Processing DeleteTablet for tablet 817012fd7566410fba9b2032900447b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41638
I20260619 16:39:38.974442 19966 tablet_service.cc:1514] Processing DeleteTablet for tablet 817012fd7566410fba9b2032900447b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37254
I20260619 16:39:38.974494 19698 tablet_service.cc:1514] Processing DeleteTablet for tablet 817012fd7566410fba9b2032900447b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42864
I20260619 16:39:38.974959 20073 tablet_replica.cc:330] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: stopping tablet replica
I20260619 16:39:38.975237 20073 raft_consensus.cc:2229] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:39:38.975558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19635
I20260619 16:39:38.976392 20073 raft_consensus.cc:2258] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:38.977094 20074 tablet_replica.cc:330] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: stopping tablet replica
I20260619 16:39:38.977391 20074 raft_consensus.cc:2229] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:38.977608 20074 raft_consensus.cc:2258] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:38.978672 20073 ts_tablet_manager.cc:1905] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:38.978876 20074 ts_tablet_manager.cc:1905] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:38.981454 20074 ts_tablet_manager.cc:1918] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:38.981454 20073 ts_tablet_manager.cc:1918] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:38.981550 20073 log.cc:1196] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/817012fd7566410fba9b2032900447b8
I20260619 16:39:38.981550 20074 log.cc:1196] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/817012fd7566410fba9b2032900447b8
I20260619 16:39:38.981876 20073 ts_tablet_manager.cc:1939] T 817012fd7566410fba9b2032900447b8 P e4e99ddc72944e8aa8ac27c9fdcfb043: Deleting consensus metadata
I20260619 16:39:38.981876 20074 ts_tablet_manager.cc:1939] T 817012fd7566410fba9b2032900447b8 P e3bddf3aa3984f65bd4c7a758dfbe832: Deleting consensus metadata
I20260619 16:39:38.982893 19560 catalog_manager.cc:4841] TS e3bddf3aa3984f65bd4c7a758dfbe832 (127.0.98.3:37849): tablet 817012fd7566410fba9b2032900447b8 (table test-table [id=d59ec658df564fa4990b24daf69a12a1]) successfully deleted
I20260619 16:39:38.982908 19559 catalog_manager.cc:4841] TS e4e99ddc72944e8aa8ac27c9fdcfb043 (127.0.98.2:44077): tablet 817012fd7566410fba9b2032900447b8 (table test-table [id=d59ec658df564fa4990b24daf69a12a1]) successfully deleted
W20260619 16:39:38.984030 19560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39313 (error 108)
I20260619 16:39:38.984802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19769
I20260619 16:39:38.984941 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:38.984672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39313 (local address 127.0.0.1:42866)
0619 16:39:38.984802 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:38.984866 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 16:39:38.985747 19560 catalog_manager.cc:4584] TS 614963eabebc4855aaa86c0d63cf3ec0 (127.0.98.1:39313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 817012fd7566410fba9b2032900447b8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39313: connect: Connection refused (error 111)
I20260619 16:39:38.989977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19903
I20260619 16:39:38.995563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 19541
2026-06-19T16:39:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2526 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T16:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:39Z Disabled control of system clock
I20260619 16:39:39.024909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42825
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:39:39.130927 20080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:39.131160 20080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:39.131196 20080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:39.135165 20080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:39.135216 20080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:39.135232 20080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:39.135248 20080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:39.135263 20080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:39.139945 20080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:39.140928 20080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:39.142246 20080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:39.150166 20086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:39.150233 20080 server_base.cc:1034] running on GCE node
W20260619 16:39:39.150158 20085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:39.150141 20088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:39.150921 20080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:39.152017 20080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:39.153193 20080 hybrid_clock.cc:648] HybridClock initialized: now 1781887179153165 us; error 37 us; skew 500 ppm
I20260619 16:39:39.155021 20080 webserver.cc:460] Webserver started at http://127.0.98.62:32813/ using document root <none> and password file <none>
I20260619 16:39:39.155598 20080 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:39.155674 20080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:39.155894 20080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:39.158041 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "6c512ee84935418e8f150efcd0e5a270"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "ae1007edd259e445ab5d978c5549f2a5"
server_key_iv: "23bca4450285c7799628813a4855ff1e"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.158619 20080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c512ee84935418e8f150efcd0e5a270"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "ae1007edd259e445ab5d978c5549f2a5"
server_key_iv: "23bca4450285c7799628813a4855ff1e"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.162345 20080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:39.164749 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.166025 20080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:39.166186 20080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "6c512ee84935418e8f150efcd0e5a270"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "ae1007edd259e445ab5d978c5549f2a5"
server_key_iv: "23bca4450285c7799628813a4855ff1e"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.166301 20080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:39.180086 20080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:39.180712 20080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:39.180904 20080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:39.187505 20080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42825
I20260619 16:39:39.187513 20146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42825 every 8 connection(s)
I20260619 16:39:39.188382 20080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:39.189937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20080
I20260619 16:39:39.190078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:39.190371 392 external_mini_cluster.cc:1428] Setting key 843a2dc7f873ce6f8177bda67f63d88f
I20260619 16:39:39.191872 20147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:39.198416 20147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270: Bootstrap starting.
I20260619 16:39:39.200199 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.191578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.0.98.62:42825)
0619 16:39:39.192195 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.192216 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.192256 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:39:39.193078 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.193097 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.193595 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.193933 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:39.194356 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.195799 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.196621 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.197302 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.197492 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.198266 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:39.198272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:39.198286 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:39.198757 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:39.198821 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:39.198846 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:39.198937 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.199069 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.199316 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":105,"threads_started":1}
I20260619 16:39:39.200965 20147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:39.201771 20147 log.cc:824] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:39.203553 20147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270: No bootstrap required, opened a new log
I20260619 16:39:39.206871 20147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c512ee84935418e8f150efcd0e5a270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42825 } }
I20260619 16:39:39.207072 20147 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:39.207129 20147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c512ee84935418e8f150efcd0e5a270, State: Initialized, Role: FOLLOWER
I20260619 16:39:39.207563 20147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c512ee84935418e8f150efcd0e5a270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42825 } }
I20260619 16:39:39.207710 20147 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:39.207775 20147 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:39.207887 20147 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:39.209111 20147 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c512ee84935418e8f150efcd0e5a270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42825 } }
I20260619 16:39:39.209537 20147 leader_election.cc:302] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c512ee84935418e8f150efcd0e5a270; no voters:
I20260619 16:39:39.209900 20147 leader_election.cc:288] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:39.210011 20152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:39.210251 20152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [term 1 LEADER]: Becoming Leader. State: Replica: 6c512ee84935418e8f150efcd0e5a270, State: Running, Role: LEADER
I20260619 16:39:39.210538 20152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c512ee84935418e8f150efcd0e5a270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42825 } }
I20260619 16:39:39.211267 20147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:39.212049 20154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c512ee84935418e8f150efcd0e5a270. Latest consensus state: current_term: 1 leader_uuid: "6c512ee84935418e8f150efcd0e5a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c512ee84935418e8f150efcd0e5a270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42825 } } }
I20260619 16:39:39.212049 20153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c512ee84935418e8f150efcd0e5a270" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c512ee84935418e8f150efcd0e5a270" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42825 } } }
I20260619 16:39:39.212203 20154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:39.212222 20153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:39.213455 20160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:39.216315 20160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:39.221813 20160 catalog_manager.cc:1347] Generated new cluster ID: 023c6acc057841deba97d8d7c49b7fbb
I20260619 16:39:39.221887 20160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:39.255635 20160 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:39.256875 20160 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:39.265282 20160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270: Generated new TSK 0
I20260619 16:39:39.266031 20160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:39.268932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42825
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 16:39:39.376495 20171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:39.376761 20171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:39.376830 20171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:39.380753 20171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:39.380828 20171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:39.380914 20171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 16:39:39.385465 20171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:39.386406 20171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:39.387696 20171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:39.396849 20177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:39.396855 20179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:39.396989 20176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:39.397039 20171 server_base.cc:1034] running on GCE node
I20260619 16:39:39.397541 20171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:39.398160 20171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:39.399323 20171 hybrid_clock.cc:648] HybridClock initialized: now 1781887179399295 us; error 39 us; skew 500 ppm
I20260619 16:39:39.401172 20171 webserver.cc:460] Webserver started at http://127.0.98.1:44611/ using document root <none> and password file <none>
I20260619 16:39:39.401736 20171 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:39.401796 20171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:39.401998 20171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:39.403905 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data/instance:
uuid: "130cc96cad224364a2a7ccb77dc06c77"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "ca2f17dc5eb40b541a9456470dc5ac70"
server_key_iv: "55f455531240c9255805d183a7c019eb"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.404446 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal/instance:
uuid: "130cc96cad224364a2a7ccb77dc06c77"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "ca2f17dc5eb40b541a9456470dc5ac70"
server_key_iv: "55f455531240c9255805d183a7c019eb"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.407847 20171 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 16:39:39.410213 20185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.411437 20171 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:39.411552 20171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal
uuid: "130cc96cad224364a2a7ccb77dc06c77"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "ca2f17dc5eb40b541a9456470dc5ac70"
server_key_iv: "55f455531240c9255805d183a7c019eb"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.411659 20171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:39.423875 20171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:39.424445 20171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:39.424623 20171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:39.425140 20171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:39.426023 20171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:39.426066 20171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.426136 20171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:39.426165 20171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.435067 20171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39731
I20260619 16:39:39.435139 20298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39731 every 8 connection(s)
I20260619 16:39:39.435907 20171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/data/info.pb
I20260619 16:39:39.445037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20171
I20260619 16:39:39.445147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-0/wal/instance
I20260619 16:39:39.445432 392 external_mini_cluster.cc:1428] Setting key e0053df6749e217e30be7c6d27ef865a
I20260619 16:39:39.447181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42825
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:39:39.449498 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.437713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50145 (local address 127.0.98.62:42825)
0619 16:39:39.437885 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.437890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.438889 (+ 999us) server_negotiation.cc:407] Connection header received
0619 16:39:39.440159 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.440162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.440211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.440287 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:39.441122 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.441801 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.442925 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.443156 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.443247 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.448917 (+ 5670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:39.448920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:39.448922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:39.448934 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:39.448953 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:39.448961 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:39.449025 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.449212 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.449270 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 16:39:39.450295 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.438079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42825 (local address 127.0.98.1:50145)
0619 16:39:39.438736 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.438773 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.439963 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.440434 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.440441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.440518 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:39.440983 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.440995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.441928 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.441931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.442789 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.442799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.443332 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.443335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.443473 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.448554 (+ 5081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:39.448557 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:39.448658 (+ 101us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:39.448747 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:39.449052 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:39.449063 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:39.449071 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.449207 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.449529 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260619 16:39:39.451467 20299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42825
I20260619 16:39:39.451694 20299 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:39.452167 20299 heartbeater.cc:511] Master 127.0.98.62:42825 requested a full tablet report, sending...
I20260619 16:39:39.453650 20111 ts_manager.cc:194] Registered new tserver with Master: 130cc96cad224364a2a7ccb77dc06c77 (127.0.98.1:39731)
I20260619 16:39:39.454771 20111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50145
W20260619 16:39:39.554611 20302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:39.554817 20302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:39.554852 20302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:39.558699 20302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:39.558740 20302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:39.558825 20302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 16:39:39.563388 20302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:39.564309 20302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:39.565601 20302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:39.575003 20310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:39.575043 20308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:39.575003 20307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:39.575332 20302 server_base.cc:1034] running on GCE node
I20260619 16:39:39.575701 20302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:39.576215 20302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:39.577399 20302 hybrid_clock.cc:648] HybridClock initialized: now 1781887179577382 us; error 25 us; skew 500 ppm
I20260619 16:39:39.579103 20302 webserver.cc:460] Webserver started at http://127.0.98.2:41159/ using document root <none> and password file <none>
I20260619 16:39:39.579639 20302 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:39.579701 20302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:39.579890 20302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:39.581820 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data/instance:
uuid: "32f09697e2bc41dd8b3aaadccb654066"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "12a0dbab663918beca5fe242dd8c5399"
server_key_iv: "70251120d4ad1ca646a6dd596d5dfbc0"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.582314 20302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32f09697e2bc41dd8b3aaadccb654066"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "12a0dbab663918beca5fe242dd8c5399"
server_key_iv: "70251120d4ad1ca646a6dd596d5dfbc0"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.585682 20302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:39:39.587914 20316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.589069 20302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:39.589161 20302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal
uuid: "32f09697e2bc41dd8b3aaadccb654066"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "12a0dbab663918beca5fe242dd8c5399"
server_key_iv: "70251120d4ad1ca646a6dd596d5dfbc0"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.589238 20302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:39.602910 20302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:39.603417 20302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:39.603546 20302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:39.604002 20302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:39.604806 20302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:39.604852 20302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.604888 20302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:39.604903 20302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.613359 20302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46733
I20260619 16:39:39.613386 20429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46733 every 8 connection(s)
I20260619 16:39:39.614174 20302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/data/info.pb
I20260619 16:39:39.622857 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.615899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60529 (local address 127.0.98.62:42825)
0619 16:39:39.616057 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.616062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.617079 (+ 1017us) server_negotiation.cc:407] Connection header received
0619 16:39:39.618510 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.618512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.618565 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.618641 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:39.619481 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.619992 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.621098 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.621308 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.621447 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.622417 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:39.622420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:39.622422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:39.622433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:39.622452 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:39.622460 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:39.622519 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.622697 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.622749 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:39:39.623214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20302
I20260619 16:39:39.623346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal/instance
I20260619 16:39:39.623657 392 external_mini_cluster.cc:1428] Setting key 388af1814c133294e075c868f7a679b3
I20260619 16:39:39.623860 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.616312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42825 (local address 127.0.98.2:60529)
0619 16:39:39.616921 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.616959 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.618314 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.618827 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.618834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.618911 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:39.619351 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.619362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.620116 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.620119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.620990 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.620999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.621514 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.621517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.621655 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.622188 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:39.622190 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:39.622224 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:39.622281 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:39.622563 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:39.622570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:39.622574 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.622680 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.622873 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260619 16:39:39.625041 20430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42825
I20260619 16:39:39.625260 20430 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:39.625679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42825
--builtin_ntp_servers=127.0.98.20:46789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 16:39:39.625881 20430 heartbeater.cc:511] Master 127.0.98.62:42825 requested a full tablet report, sending...
I20260619 16:39:39.626907 20111 ts_manager.cc:194] Registered new tserver with Master: 32f09697e2bc41dd8b3aaadccb654066 (127.0.98.2:46733)
I20260619 16:39:39.627453 20111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60529
W20260619 16:39:39.731034 20433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:39.731252 20433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:39.731288 20433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:39.735229 20433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:39.735275 20433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:39.735325 20433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 16:39:39.739841 20433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:39.740823 20433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:39.742262 20433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:39.751578 20438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:39.751675 20433 server_base.cc:1034] running on GCE node
W20260619 16:39:39.751602 20441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:39.751602 20439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:39.752162 20433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:39.752704 20433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:39.753902 20433 hybrid_clock.cc:648] HybridClock initialized: now 1781887179753876 us; error 40 us; skew 500 ppm
I20260619 16:39:39.755656 20433 webserver.cc:460] Webserver started at http://127.0.98.3:45519/ using document root <none> and password file <none>
I20260619 16:39:39.756225 20433 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:39.756315 20433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:39.756531 20433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:39.758515 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data/instance:
uuid: "be3fd68f6dc84504b20d093ff97559d4"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "cfee53405d689064871ad77da04b7a34"
server_key_iv: "2d70530f585f77a9fb8983ab52ad7f21"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.759048 20433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal/instance:
uuid: "be3fd68f6dc84504b20d093ff97559d4"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "cfee53405d689064871ad77da04b7a34"
server_key_iv: "2d70530f585f77a9fb8983ab52ad7f21"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.762499 20433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:39.764757 20447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.765934 20433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:39.766068 20433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal
uuid: "be3fd68f6dc84504b20d093ff97559d4"
format_stamp: "Formatted at 2026-06-19 16:39:39 on dist-test-slave-0f28"
server_key: "cfee53405d689064871ad77da04b7a34"
server_key_iv: "2d70530f585f77a9fb8983ab52ad7f21"
server_key_version: "encryptionkey@0"
I20260619 16:39:39.766181 20433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:39.791378 20433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:39.791967 20433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:39.792152 20433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:39.792729 20433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 16:39:39.793589 20433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 16:39:39.793658 20433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.793726 20433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 16:39:39.793777 20433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:39.801952 20433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39625
I20260619 16:39:39.801975 20560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39625 every 8 connection(s)
I20260619 16:39:39.802747 20433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/data/info.pb
I20260619 16:39:39.811421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20433
I20260619 16:39:39.811539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal/instance
I20260619 16:39:39.811632 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.804427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38247 (local address 127.0.98.62:42825)
0619 16:39:39.804553 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.804557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.805613 (+ 1056us) server_negotiation.cc:407] Connection header received
0619 16:39:39.807010 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.807013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.807071 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.807191 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:39.807992 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.808523 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.809741 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.809963 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.810073 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.811118 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:39.811127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:39.811129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:39.811139 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:39.811159 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:39.811167 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:39.811265 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.811428 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.811486 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 16:39:39.811869 392 external_mini_cluster.cc:1428] Setting key e5c4796a7742ba4ead30fd578a61501e
I20260619 16:39:39.812595 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.804785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42825 (local address 127.0.98.3:38247)
0619 16:39:39.805476 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.805513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.806808 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.807340 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.807346 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.807422 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:39:39.807854 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.807865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.808651 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.808665 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.809610 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.809619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.810115 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.810118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.810278 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.810859 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 16:39:39.810861 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:39:39.810897 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 16:39:39.810962 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:39:39.811266 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:39:39.811277 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:39:39.811284 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.811411 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.811717 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260619 16:39:39.813673 20561 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42825
I20260619 16:39:39.813906 20561 heartbeater.cc:464] Registering TS with master...
I20260619 16:39:39.814384 20561 heartbeater.cc:511] Master 127.0.98.62:42825 requested a full tablet report, sending...
I20260619 16:39:39.815418 20111 ts_manager.cc:194] Registered new tserver with Master: be3fd68f6dc84504b20d093ff97559d4 (127.0.98.3:39625)
I20260619 16:39:39.816134 20111 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38247
I20260619 16:39:39.825801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 16:39:39.831583 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.827784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.0.98.62:42825)
0619 16:39:39.827947 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.827950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.828056 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:39:39.828121 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.828123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.828161 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.828229 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:39.828503 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.829031 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.829980 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.830189 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.830270 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.830935 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:39.830942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:39.830946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:39.830961 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:39.830986 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:39.831002 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:39.831082 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.831313 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.831392 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 16:39:39.834548 20111 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 16:39:39.837430 20111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 16:39:39.850930 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.845451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39731 (local address 127.0.0.1:48150)
0619 16:39:39.846295 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.846340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.846456 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.847338 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.847343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.847365 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:39.847545 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.847551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.848516 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.848537 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.849716 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.849725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.850536 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.850538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.850645 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.850673 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.850742 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.850820 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":71,"spinlock_wait_cycles":6144,"thread_start_us":123,"threads_started":1}
I20260619 16:39:39.851032 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.845712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.0.98.1:39731)
0619 16:39:39.846703 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.846715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.846739 (+ 24us) server_negotiation.cc:407] Connection header received
0619 16:39:39.846818 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.846824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.847019 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.847185 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:39.847680 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.848399 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.849840 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.850424 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.850533 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.850719 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.850783 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.850872 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":117,"threads_started":1}
I20260619 16:39:39.851668 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.845517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39625 (local address 127.0.0.1:44868)
0619 16:39:39.846444 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.846457 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.846558 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.847191 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.847197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.847217 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:39.847357 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.847365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.848813 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.848818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.850265 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.850275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.851312 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.851316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.851408 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.851431 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.851479 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.851542 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":120,"thread_start_us":93,"threads_started":1}
I20260619 16:39:39.852329 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.846076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.0.98.3:39625)
0619 16:39:39.846800 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.846810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.846827 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:39:39.846888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.846892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.847031 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.847185 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:39.847486 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.848672 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.850404 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.851174 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.851834 (+ 660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.851999 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.852060 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.852144 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":128,"threads_started":1}
I20260619 16:39:39.852648 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.846204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46733 (local address 127.0.0.1:48314)
0619 16:39:39.846664 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.846678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.846755 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.847371 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.847373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.847394 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:39.847532 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.847541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.848857 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.848859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.850265 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.850275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.852262 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.852266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.852357 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.852382 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.852437 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.852526 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":176,"spinlock_wait_cycles":384,"thread_start_us":83,"threads_started":1}
I20260619 16:39:39.853914 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.846603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.98.2:46733)
0619 16:39:39.846962 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.846970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.846987 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:39:39.847045 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.847048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.847243 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.847389 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:39.847656 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.848746 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.851289 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.852124 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.853444 (+ 1320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.853628 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.853677 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.853761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260619 16:39:39.853863 20233 tablet_service.cc:1467] Processing CreateTablet for tablet 6739457fe86d477389b6b0c7733d388b (DEFAULT_TABLE table=test-table [id=96177b9f706e4d308f368c292b0ba2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:39.854952 20233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6739457fe86d477389b6b0c7733d388b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:39.855190 20495 tablet_service.cc:1467] Processing CreateTablet for tablet 6739457fe86d477389b6b0c7733d388b (DEFAULT_TABLE table=test-table [id=96177b9f706e4d308f368c292b0ba2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:39.856119 20495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6739457fe86d477389b6b0c7733d388b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:39.856375 20364 tablet_service.cc:1467] Processing CreateTablet for tablet 6739457fe86d477389b6b0c7733d388b (DEFAULT_TABLE table=test-table [id=96177b9f706e4d308f368c292b0ba2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 16:39:39.857301 20364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6739457fe86d477389b6b0c7733d388b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:39.861625 20578 tablet_bootstrap.cc:492] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Bootstrap starting.
I20260619 16:39:39.862921 20579 tablet_bootstrap.cc:492] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: Bootstrap starting.
I20260619 16:39:39.863817 20578 tablet_bootstrap.cc:654] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:39.864612 20580 tablet_bootstrap.cc:492] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Bootstrap starting.
I20260619 16:39:39.864655 20578 log.cc:824] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:39.864966 20579 tablet_bootstrap.cc:654] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:39.865793 20579 log.cc:824] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:39.866802 20580 tablet_bootstrap.cc:654] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:39.866822 20578 tablet_bootstrap.cc:492] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: No bootstrap required, opened a new log
I20260619 16:39:39.867000 20578 ts_tablet_manager.cc:1397] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 16:39:39.867508 20580 log.cc:824] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:39.867611 20579 tablet_bootstrap.cc:492] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: No bootstrap required, opened a new log
I20260619 16:39:39.867779 20579 ts_tablet_manager.cc:1397] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 16:39:39.869017 20580 tablet_bootstrap.cc:492] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: No bootstrap required, opened a new log
I20260619 16:39:39.869211 20580 ts_tablet_manager.cc:1397] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 16:39:39.871272 20578 raft_consensus.cc:348] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.871529 20578 raft_consensus.cc:374] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:39.871588 20578 raft_consensus.cc:729] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3fd68f6dc84504b20d093ff97559d4, State: Initialized, Role: FOLLOWER
I20260619 16:39:39.871831 20579 raft_consensus.cc:348] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.872035 20579 raft_consensus.cc:374] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:39.872081 20579 raft_consensus.cc:729] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130cc96cad224364a2a7ccb77dc06c77, State: Initialized, Role: FOLLOWER
I20260619 16:39:39.872044 20578 consensus_queue.cc:260] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.872507 20579 consensus_queue.cc:260] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.872998 20578 ts_tablet_manager.cc:1428] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 16:39:39.873008 20580 raft_consensus.cc:348] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.873272 20580 raft_consensus.cc:374] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:39.873322 20580 raft_consensus.cc:729] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f09697e2bc41dd8b3aaadccb654066, State: Initialized, Role: FOLLOWER
I20260619 16:39:39.873384 20579 ts_tablet_manager.cc:1428] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 16:39:39.873399 20561 heartbeater.cc:503] Master 127.0.98.62:42825 was elected leader, sending a full tablet report...
I20260619 16:39:39.873723 20299 heartbeater.cc:503] Master 127.0.98.62:42825 was elected leader, sending a full tablet report...
I20260619 16:39:39.873724 20580 consensus_queue.cc:260] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.874612 20580 ts_tablet_manager.cc:1428] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 16:39:39.874608 20430 heartbeater.cc:503] Master 127.0.98.62:42825 was elected leader, sending a full tablet report...
W20260619 16:39:39.937016 20300 tablet.cc:2369] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:39:39.979514 20584 raft_consensus.cc:482] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 16:39:39.979707 20584 raft_consensus.cc:504] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.980829 20584 leader_election.cc:288] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 130cc96cad224364a2a7ccb77dc06c77 (127.0.98.1:39731), 32f09697e2bc41dd8b3aaadccb654066 (127.0.98.2:46733)
I20260619 16:39:39.986404 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.981525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60705 (local address 127.0.98.2:46733)
0619 16:39:39.981691 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.981696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.981883 (+ 187us) server_negotiation.cc:407] Connection header received
0619 16:39:39.982037 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.982041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.982095 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.982187 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:39.982551 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.983369 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.985081 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.985706 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.985814 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.985873 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.986209 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.986278 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 16:39:39.986405 20574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.981156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34889 (local address 127.0.98.1:39731)
0619 16:39:39.981282 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:39.981285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:39.981300 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:39:39.981424 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:39.981430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:39.981486 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:39.981595 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 16:39:39.982650 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.983369 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.985081 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.985696 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.985814 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.985873 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:39.986209 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:39:39.986277 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:39:39.986434 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.981056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39731 (local address 127.0.98.3:34889)
0619 16:39:39.981190 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.981206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.981301 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.981659 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.981664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.981685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:39.981811 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.981821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.983494 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.983498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.984938 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.984950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.985849 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.985853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.986051 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.986089 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.986174 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.986272 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":222}
I20260619 16:39:39.986441 20587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:39.981396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46733 (local address 127.0.98.3:60705)
0619 16:39:39.981796 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:39.981808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:39:39.981914 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:39:39.982229 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:39:39.982234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:39:39.982251 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 16:39:39.982399 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.982405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.983494 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.983498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.984938 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:39:39.984950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:39:39.985849 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:39:39.985853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:39:39.986046 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:39.986100 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 16:39:39.986173 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 16:39:39.986272 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":122,"thread_start_us":136,"threads_started":1}
I20260619 16:39:39.987118 20253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739457fe86d477389b6b0c7733d388b" candidate_uuid: "be3fd68f6dc84504b20d093ff97559d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130cc96cad224364a2a7ccb77dc06c77" is_pre_election: true
I20260619 16:39:39.987128 20384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739457fe86d477389b6b0c7733d388b" candidate_uuid: "be3fd68f6dc84504b20d093ff97559d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f09697e2bc41dd8b3aaadccb654066" is_pre_election: true
I20260619 16:39:39.987371 20384 raft_consensus.cc:2454] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be3fd68f6dc84504b20d093ff97559d4 in term 0.
I20260619 16:39:39.987413 20253 raft_consensus.cc:2454] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be3fd68f6dc84504b20d093ff97559d4 in term 0.
I20260619 16:39:39.987808 20450 leader_election.cc:302] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f09697e2bc41dd8b3aaadccb654066, be3fd68f6dc84504b20d093ff97559d4; no voters:
I20260619 16:39:39.988039 20584 raft_consensus.cc:2781] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 16:39:39.988122 20584 raft_consensus.cc:482] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 16:39:39.988166 20584 raft_consensus.cc:3037] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:39.989117 20584 raft_consensus.cc:504] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.989441 20584 leader_election.cc:288] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [CANDIDATE]: Term 1 election: Requested vote from peers 130cc96cad224364a2a7ccb77dc06c77 (127.0.98.1:39731), 32f09697e2bc41dd8b3aaadccb654066 (127.0.98.2:46733)
I20260619 16:39:39.989734 20253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739457fe86d477389b6b0c7733d388b" candidate_uuid: "be3fd68f6dc84504b20d093ff97559d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130cc96cad224364a2a7ccb77dc06c77"
I20260619 16:39:39.989842 20384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6739457fe86d477389b6b0c7733d388b" candidate_uuid: "be3fd68f6dc84504b20d093ff97559d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f09697e2bc41dd8b3aaadccb654066"
I20260619 16:39:39.989856 20253 raft_consensus.cc:3037] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:39.989979 20384 raft_consensus.cc:3037] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:39.990898 20253 raft_consensus.cc:2454] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be3fd68f6dc84504b20d093ff97559d4 in term 1.
I20260619 16:39:39.990931 20384 raft_consensus.cc:2454] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be3fd68f6dc84504b20d093ff97559d4 in term 1.
I20260619 16:39:39.991183 20450 leader_election.cc:302] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130cc96cad224364a2a7ccb77dc06c77, be3fd68f6dc84504b20d093ff97559d4; no voters:
I20260619 16:39:39.991353 20584 raft_consensus.cc:2781] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:39.991573 20584 raft_consensus.cc:686] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 1 LEADER]: Becoming Leader. State: Replica: be3fd68f6dc84504b20d093ff97559d4, State: Running, Role: LEADER
I20260619 16:39:39.991881 20584 consensus_queue.cc:237] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } }
I20260619 16:39:39.995152 20111 catalog_manager.cc:5495] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to be3fd68f6dc84504b20d093ff97559d4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "be3fd68f6dc84504b20d093ff97559d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be3fd68f6dc84504b20d093ff97559d4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 } health_report { overall_health: UNKNOWN } } }
I20260619 16:39:40.007036 20575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:40.003492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.98.3:39625)
0619 16:39:40.003616 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:40.003624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:40.003636 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:39:40.003761 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:40.003763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:40.003809 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:40.003894 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:39:40.004138 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:40.004617 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:40.005654 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:40.005864 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:40.005958 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:40.006287 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:39:40.006691 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:39:40.006782 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:40.006871 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 16:39:40.006914 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 16:39:40.012646 20384 raft_consensus.cc:1261] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 1 FOLLOWER]: Refusing update from remote peer be3fd68f6dc84504b20d093ff97559d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:40.012660 20253 raft_consensus.cc:1261] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77 [term 1 FOLLOWER]: Refusing update from remote peer be3fd68f6dc84504b20d093ff97559d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:39:40.013353 20584 consensus_queue.cc:1035] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32f09697e2bc41dd8b3aaadccb654066" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:40.013607 20588 consensus_queue.cc:1035] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130cc96cad224364a2a7ccb77dc06c77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:39:40.023664 20593 mvcc.cc:205] Tried to move back new op lower bound from 7298609889324613632 to 7298609889249689600. Current Snapshot: MvccSnapshot[applied={T|T < 7298609889324613632}]
I20260619 16:39:40.028292 20109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:40.028461 20109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 16:39:40.030758 20109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c512ee84935418e8f150efcd0e5a270: Sending DeleteTablet for 3 replicas of tablet 6739457fe86d477389b6b0c7733d388b
I20260619 16:39:40.031529 20233 tablet_service.cc:1514] Processing DeleteTablet for tablet 6739457fe86d477389b6b0c7733d388b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:40 UTC) from {username='slave'} at 127.0.0.1:48150
I20260619 16:39:40.031750 20495 tablet_service.cc:1514] Processing DeleteTablet for tablet 6739457fe86d477389b6b0c7733d388b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:40 UTC) from {username='slave'} at 127.0.0.1:44868
I20260619 16:39:40.031725 20364 tablet_service.cc:1514] Processing DeleteTablet for tablet 6739457fe86d477389b6b0c7733d388b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 16:39:40 UTC) from {username='slave'} at 127.0.0.1:48314
I20260619 16:39:40.032215 20602 tablet_replica.cc:330] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: stopping tablet replica
I20260619 16:39:40.032544 20602 raft_consensus.cc:2229] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 16:39:40.032770 20601 tablet_replica.cc:330] T 6739457fe86d477389b6b0c7733d388b P 130cc96cad224364a2a7ccb77dc06c77: stopping tablet replica
I20260619 16:39:40.032830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20171
I20260619 16:39:40.032882 20602 raft_consensus.cc:2258] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:40.033233 20603 tablet_replica.cc:330] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: stopping tablet replica
I20260619 16:39:40.033583 20603 raft_consensus.cc:2229] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 16:39:40.033856 20603 raft_consensus.cc:2258] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 16:39:40.035287 20603 ts_tablet_manager.cc:1905] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:40.035290 20602 ts_tablet_manager.cc:1905] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 16:39:40.037958 20602 ts_tablet_manager.cc:1918] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:40.037951 20603 ts_tablet_manager.cc:1918] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 16:39:40.038072 20602 log.cc:1196] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-2/wal/wals/6739457fe86d477389b6b0c7733d388b
I20260619 16:39:40.038079 20603 log.cc:1196] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Deleting WAL directory at /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781886927079793-392-0/minicluster-data/ts-1/wal/wals/6739457fe86d477389b6b0c7733d388b
I20260619 16:39:40.038357 20602 ts_tablet_manager.cc:1939] T 6739457fe86d477389b6b0c7733d388b P be3fd68f6dc84504b20d093ff97559d4: Deleting consensus metadata
I20260619 16:39:40.038374 20603 ts_tablet_manager.cc:1939] T 6739457fe86d477389b6b0c7733d388b P 32f09697e2bc41dd8b3aaadccb654066: Deleting consensus metadata
I20260619 16:39:40.039354 20096 catalog_manager.cc:4841] TS be3fd68f6dc84504b20d093ff97559d4 (127.0.98.3:39625): tablet 6739457fe86d477389b6b0c7733d388b (table test-table [id=96177b9f706e4d308f368c292b0ba2c5]) successfully deleted
I20260619 16:39:40.039413 20097 catalog_manager.cc:4841] TS 32f09697e2bc41dd8b3aaadccb654066 (127.0.98.2:46733): tablet 6739457fe86d477389b6b0c7733d388b (table test-table [id=96177b9f706e4d308f368c292b0ba2c5]) successfully deleted
W20260619 16:39:40.043311 20097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39731 (error 108)
I20260619 16:39:40.044039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20302
I20260619 16:39:40.044315 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:40.044021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39731 (local address 127.0.0.1:48152)
0619 16:39:40.044149 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:39:40.044220 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 16:39:40.045568 20097 catalog_manager.cc:4584] TS 130cc96cad224364a2a7ccb77dc06c77 (127.0.98.1:39731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6739457fe86d477389b6b0c7733d388b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39731: connect: Connection refused (error 111)
I20260619 16:39:40.050874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20433
I20260619 16:39:40.056867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20080
2026-06-19T16:39:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1060 ms)
[----------] 6 tests from EncryptionPolicyTest (10413 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:40 dist-test-slave-0f28 krb5kdc[20607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:40 dist-test-slave-0f28 krb5kdc[20607](info): set up 2 sockets
Jun 19 16:39:40 dist-test-slave-0f28 krb5kdc[20607](info): commencing operation
krb5kdc: starting...
W20260619 16:39:41.137538 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:41 dist-test-slave-0f28 krb5kdc[20607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:41.260452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:41.364869 20620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:41.365152 20620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:41.365248 20620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:41.369164 20620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:41.369247 20620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:41.369294 20620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:41.369370 20620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:41.369401 20620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:41.374567 20620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:41.375557 20620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:41.376528 20620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:41.384769 20626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:41.384774 20625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:41.384974 20620 server_base.cc:1034] running on GCE node
W20260619 16:39:41.384830 20628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:41.385725 20620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:41.386747 20620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:41.387945 20620 hybrid_clock.cc:648] HybridClock initialized: now 1781887181387926 us; error 38 us; skew 500 ppm
Jun 19 16:39:41 dist-test-slave-0f28 krb5kdc[20607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:41.390704 20620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:41.391518 20620 webserver.cc:460] Webserver started at http://127.0.0.1:32959/ using document root <none> and password file <none>
I20260619 16:39:41.392060 20620 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:41.392175 20620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:41.392397 20620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:41.394424 20620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "9fc09a1aca8540cc837841ecd067c3a9"
format_stamp: "Formatted at 2026-06-19 16:39:41 on dist-test-slave-0f28"
server_key: "21f0241c9de68f13482d20ee7ba89484"
server_key_iv: "a9a97dc60956191c8ce5a097b7e2d2bc"
server_key_version: "encryptionkey@0"
I20260619 16:39:41.394966 20620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "9fc09a1aca8540cc837841ecd067c3a9"
format_stamp: "Formatted at 2026-06-19 16:39:41 on dist-test-slave-0f28"
server_key: "21f0241c9de68f13482d20ee7ba89484"
server_key_iv: "a9a97dc60956191c8ce5a097b7e2d2bc"
server_key_version: "encryptionkey@0"
I20260619 16:39:41.398547 20620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:41.400774 20635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:41.402055 20620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:41.402196 20620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "9fc09a1aca8540cc837841ecd067c3a9"
format_stamp: "Formatted at 2026-06-19 16:39:41 on dist-test-slave-0f28"
server_key: "21f0241c9de68f13482d20ee7ba89484"
server_key_iv: "a9a97dc60956191c8ce5a097b7e2d2bc"
server_key_version: "encryptionkey@0"
I20260619 16:39:41.402308 20620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:41.414559 20620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:41.417454 20620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:41.417651 20620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:41.424396 20620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33327
I20260619 16:39:41.424408 20687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33327 every 8 connection(s)
I20260619 16:39:41.425347 20620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:41.425803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20620
I20260619 16:39:41.425954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:41.426249 392 external_mini_cluster.cc:1428] Setting key 0bda0e36b7cca53962070ac45182beae
I20260619 16:39:41.428508 20688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:41.434778 20688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9: Bootstrap starting.
Jun 19 16:39:41 dist-test-slave-0f28 krb5kdc[20607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:39:41.437282 20688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:41.438171 20688 log.cc:824] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:41.440415 20688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9: No bootstrap required, opened a new log
I20260619 16:39:41.441161 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:41.427648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54111 (local address 127.0.0.1:33327)
0619 16:39:41.428070 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:41.428100 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:41.428303 (+ 203us) server_negotiation.cc:407] Connection header received
0619 16:39:41.429103 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:41.429120 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:41.429510 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:41.429775 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:41.430665 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:41.431646 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:41.432794 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:41.433232 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:41.433407 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:41.436060 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:41.436065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:41.436080 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:41.436108 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:41.438013 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:41.438566 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:41.438575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:41.438580 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:41.438688 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:41.439097 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:41.439104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:41.439107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:41.439588 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:41.439786 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:41.440080 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 16:39:41.440293 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260619 16:39:41.444048 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:41.444492 20688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc09a1aca8540cc837841ecd067c3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260619 16:39:41.444689 20688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:41.444766 20688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc09a1aca8540cc837841ecd067c3a9, State: Initialized, Role: FOLLOWER
I20260619 16:39:41.445205 20688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc09a1aca8540cc837841ecd067c3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260619 16:39:41.445420 20688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:41.445514 20688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:41.445614 20688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:41.446821 20688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc09a1aca8540cc837841ecd067c3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260619 16:39:41.447201 20688 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc09a1aca8540cc837841ecd067c3a9; no voters:
I20260619 16:39:41.447532 20688 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:41.447644 20698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:41.447814 20698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [term 1 LEADER]: Becoming Leader. State: Replica: 9fc09a1aca8540cc837841ecd067c3a9, State: Running, Role: LEADER
I20260619 16:39:41.448149 20698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc09a1aca8540cc837841ecd067c3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } }
I20260619 16:39:41.448710 20688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:41.449303 20698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc09a1aca8540cc837841ecd067c3a9. Latest consensus state: current_term: 1 leader_uuid: "9fc09a1aca8540cc837841ecd067c3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc09a1aca8540cc837841ecd067c3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } } }
I20260619 16:39:41.450122 20698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:41.450012 20699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc09a1aca8540cc837841ecd067c3a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc09a1aca8540cc837841ecd067c3a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33327 } } }
I20260619 16:39:41.450443 20699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:41.450805 20707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:41.453959 20707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:41.455240 20691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:41.446123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49523 (local address 127.0.0.1:33327)
0619 16:39:41.446225 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:41.446229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:41.446412 (+ 183us) server_negotiation.cc:407] Connection header received
0619 16:39:41.446524 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:41.446526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:41.446558 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:41.446630 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:41.447434 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:41.448105 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:41.448913 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:41.449185 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:41.449352 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:41.451048 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:41.451054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:41.451059 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:41.451102 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:41.452867 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:41.453477 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:41.453481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:41.453483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:41.453564 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:41.453928 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:41.453931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:41.453933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:41.454139 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:41.454274 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:41.454986 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 16:39:41.455078 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 16:39:41.459615 20707 catalog_manager.cc:1347] Generated new cluster ID: 1ad4797f9118442bb0403a50340e30a8
I20260619 16:39:41.459677 20707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:41.467903 20707 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:41.469013 20707 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:41.475133 20707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc09a1aca8540cc837841ecd067c3a9: Generated new TSK 0
I20260619 16:39:41.475881 20707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:41.481063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20620
W20260619 16:39:41.485898 20605 connection.cc:535] client connection to 127.0.0.1:33327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33327): Connection reset by peer (error 104)
2026-06-19T16:39:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:41 dist-test-slave-0f28 krb5kdc[20719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:41 dist-test-slave-0f28 krb5kdc[20719](info): set up 2 sockets
Jun 19 16:39:41 dist-test-slave-0f28 krb5kdc[20719](info): commencing operation
krb5kdc: starting...
W20260619 16:39:42.521574 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:42 dist-test-slave-0f28 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:42.637045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:42.741971 20732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:42.742241 20732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:42.742309 20732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:42.746332 20732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:42.746407 20732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:42.746455 20732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:42.746495 20732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:42.746528 20732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:42.751613 20732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:42.752682 20732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:42.753676 20732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:42.762019 20738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:42.762007 20737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:42.762068 20732 server_base.cc:1034] running on GCE node
W20260619 16:39:42.762020 20740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:42.762671 20732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:42.763746 20732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:42.764930 20732 hybrid_clock.cc:648] HybridClock initialized: now 1781887182764903 us; error 41 us; skew 500 ppm
Jun 19 16:39:42 dist-test-slave-0f28 krb5kdc[20719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:42.767627 20732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:42.768455 20732 webserver.cc:460] Webserver started at http://127.0.0.1:38073/ using document root <none> and password file <none>
I20260619 16:39:42.768970 20732 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:42.769053 20732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:42.769299 20732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:42.771243 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "8d64e61dab9d411baed0cb0c0236b0dd"
format_stamp: "Formatted at 2026-06-19 16:39:42 on dist-test-slave-0f28"
server_key: "43a0dd7bd9bfdee09c31174f5fcc5d84"
server_key_iv: "58f4c49676da5da5574a0c32bc7054ad"
server_key_version: "encryptionkey@0"
I20260619 16:39:42.771765 20732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d64e61dab9d411baed0cb0c0236b0dd"
format_stamp: "Formatted at 2026-06-19 16:39:42 on dist-test-slave-0f28"
server_key: "43a0dd7bd9bfdee09c31174f5fcc5d84"
server_key_iv: "58f4c49676da5da5574a0c32bc7054ad"
server_key_version: "encryptionkey@0"
I20260619 16:39:42.775259 20732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:42.777614 20747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:42.778753 20732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:42.778882 20732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "8d64e61dab9d411baed0cb0c0236b0dd"
format_stamp: "Formatted at 2026-06-19 16:39:42 on dist-test-slave-0f28"
server_key: "43a0dd7bd9bfdee09c31174f5fcc5d84"
server_key_iv: "58f4c49676da5da5574a0c32bc7054ad"
server_key_version: "encryptionkey@0"
I20260619 16:39:42.778986 20732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:42.793627 20732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:42.796290 20732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:42.796475 20732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:42.803597 20732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44717
I20260619 16:39:42.803634 20799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44717 every 8 connection(s)
I20260619 16:39:42.804625 20732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:42.807940 20800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:42.812319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20732
I20260619 16:39:42.812500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:42.812791 392 external_mini_cluster.cc:1428] Setting key 698af751f395f4cab61b3d6575e677ae
I20260619 16:39:42.814240 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd: Bootstrap starting.
I20260619 16:39:42.816457 20800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:42.817153 20800 log.cc:824] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:42.819109 20800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd: No bootstrap required, opened a new log
I20260619 16:39:42.821609 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:42.814155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57979 (local address 127.0.0.1:44717)
0619 16:39:42.814555 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:42.814578 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:42.814612 (+ 34us) server_negotiation.cc:407] Connection header received
0619 16:39:42.815229 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:42.815244 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:42.815563 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:42.815827 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:42.816141 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:42.817153 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:42.818281 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:42.818743 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:42.818859 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:42.819931 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:42.819936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:42.819947 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:42.820312 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:42.820364 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:42.820385 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:42.820521 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:42.820700 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:39:42.820837 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260619 16:39:42.822784 20800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44717 } }
I20260619 16:39:42.822975 20800 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:42.823027 20800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d64e61dab9d411baed0cb0c0236b0dd, State: Initialized, Role: FOLLOWER
I20260619 16:39:42.823468 20800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44717 } }
I20260619 16:39:42.823648 20800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:42.823719 20800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:42.823814 20800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:42.824290 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:42.824803 20800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44717 } }
I20260619 16:39:42.825155 20800 leader_election.cc:302] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d64e61dab9d411baed0cb0c0236b0dd; no voters:
I20260619 16:39:42.825524 20800 leader_election.cc:288] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:42.825618 20805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:42.825847 20805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [term 1 LEADER]: Becoming Leader. State: Replica: 8d64e61dab9d411baed0cb0c0236b0dd, State: Running, Role: LEADER
I20260619 16:39:42.826146 20805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44717 } }
I20260619 16:39:42.826938 20800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:42.827270 20805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44717 } } }
I20260619 16:39:42.827392 20805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:42.827733 20810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d64e61dab9d411baed0cb0c0236b0dd. Latest consensus state: current_term: 1 leader_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d64e61dab9d411baed0cb0c0236b0dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44717 } } }
I20260619 16:39:42.827812 20810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:42.831200 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:42.826427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58583 (local address 127.0.0.1:44717)
0619 16:39:42.826517 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:42.826521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:42.826794 (+ 273us) server_negotiation.cc:407] Connection header received
0619 16:39:42.826909 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:42.826912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:42.826953 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:42.827005 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:42.827441 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:42.828239 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:42.829515 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:42.829828 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:42.829931 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:42.830757 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:42.830760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:42.830763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:42.830776 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:42.830800 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:42.830812 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:42.830899 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:42.831049 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 16:39:42.831080 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 16:39:42.832199 20825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:39:42.832281 20825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:39:42.832814 20814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:42.833617 20814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:42.838411 20814 catalog_manager.cc:1347] Generated new cluster ID: fb65e59c84ad4416bdd445f8039140da
I20260619 16:39:42.838469 20814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:42.845728 20814 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:42.846558 20814 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:42.853075 20814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d64e61dab9d411baed0cb0c0236b0dd: Generated new TSK 0
I20260619 16:39:42.853745 20814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:42.858500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20732
W20260619 16:39:42.865146 20717 connection.cc:535] client connection to 127.0.0.1:44717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44717): Connection reset by peer (error 104)
2026-06-19T16:39:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:42 dist-test-slave-0f28 krb5kdc[20831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:42 dist-test-slave-0f28 krb5kdc[20831](info): set up 2 sockets
Jun 19 16:39:42 dist-test-slave-0f28 krb5kdc[20831](info): commencing operation
krb5kdc: starting...
W20260619 16:39:43.921804 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:44 dist-test-slave-0f28 krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:44.041850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:44.150417 20844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:44.150802 20844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:44.150990 20844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:44.159098 20844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:44.159192 20844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:44.159227 20844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:44.159256 20844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:44.159288 20844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:44.165869 20844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:44.166862 20844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:44.167807 20844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:44.175876 20850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:44.175866 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:44.175866 20849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:44.176321 20844 server_base.cc:1034] running on GCE node
I20260619 16:39:44.176851 20844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:44.178037 20844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:44.179215 20844 hybrid_clock.cc:648] HybridClock initialized: now 1781887184179189 us; error 39 us; skew 500 ppm
Jun 19 16:39:44 dist-test-slave-0f28 krb5kdc[20831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:44.182241 20844 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:44.183193 20844 webserver.cc:460] Webserver started at http://127.0.0.1:36383/ using document root <none> and password file <none>
I20260619 16:39:44.183770 20844 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:44.183861 20844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:44.184093 20844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:44.186286 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "df366a638ad848bb8dec9c8748db8d0b"
format_stamp: "Formatted at 2026-06-19 16:39:44 on dist-test-slave-0f28"
server_key: "73e73e424879d1345e901fa16f6cde53"
server_key_iv: "3bfdb72dd231526fe5b57608632bc767"
server_key_version: "encryptionkey@0"
I20260619 16:39:44.186904 20844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "df366a638ad848bb8dec9c8748db8d0b"
format_stamp: "Formatted at 2026-06-19 16:39:44 on dist-test-slave-0f28"
server_key: "73e73e424879d1345e901fa16f6cde53"
server_key_iv: "3bfdb72dd231526fe5b57608632bc767"
server_key_version: "encryptionkey@0"
I20260619 16:39:44.190609 20844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:44.193128 20859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:44.194545 20844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 16:39:44.194689 20844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "df366a638ad848bb8dec9c8748db8d0b"
format_stamp: "Formatted at 2026-06-19 16:39:44 on dist-test-slave-0f28"
server_key: "73e73e424879d1345e901fa16f6cde53"
server_key_iv: "3bfdb72dd231526fe5b57608632bc767"
server_key_version: "encryptionkey@0"
I20260619 16:39:44.194800 20844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:44.199679 20844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:44.202936 20844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:44.203118 20844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:44.210156 20844 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36817
I20260619 16:39:44.210178 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36817 every 8 connection(s)
I20260619 16:39:44.211017 20844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:44.213562 20912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:44.217116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20844
I20260619 16:39:44.217259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:44.217689 392 external_mini_cluster.cc:1428] Setting key 59cd14686253fb1e74ba358b4546f479
I20260619 16:39:44.219393 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b: Bootstrap starting.
I20260619 16:39:44.221719 20912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:44.222502 20912 log.cc:824] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:44.222684 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:44.219039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55347 (local address 127.0.0.1:36817)
0619 16:39:44.219468 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:44.219486 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:44.219652 (+ 166us) server_negotiation.cc:407] Connection header received
0619 16:39:44.220269 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:44.220283 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:44.220537 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:44.220851 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:44.221591 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:44.221600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:44.221611 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:44.221621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:44.221668 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:44.221686 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:44.221792 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:44.221891 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 16:39:44.222044 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":127,"threads_started":1}
I20260619 16:39:44.224267 20912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b: No bootstrap required, opened a new log
I20260619 16:39:44.225072 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:44.228310 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:44.226992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41167 (local address 127.0.0.1:36817)
0619 16:39:44.227135 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:44.227141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:44.227189 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:39:44.227293 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:44.227296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:44.227327 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:44.227419 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:44.227979 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:44.227983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:44.227985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:44.227997 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:44.228032 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:44.228044 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:44.228134 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:44.228172 (+ 38us) server_negotiation.cc:299] Negotiation successful
0619 16:39:44.228209 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 16:39:44.228179 20912 raft_consensus.cc:348] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df366a638ad848bb8dec9c8748db8d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260619 16:39:44.228420 20912 raft_consensus.cc:374] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:44.228482 20912 raft_consensus.cc:729] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df366a638ad848bb8dec9c8748db8d0b, State: Initialized, Role: FOLLOWER
I20260619 16:39:44.228886 20912 consensus_queue.cc:260] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df366a638ad848bb8dec9c8748db8d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260619 16:39:44.229003 20912 raft_consensus.cc:388] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:44.229087 20912 raft_consensus.cc:482] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:44.229187 20912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:44.230038 20912 raft_consensus.cc:504] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df366a638ad848bb8dec9c8748db8d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260619 16:39:44.230350 20912 leader_election.cc:302] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df366a638ad848bb8dec9c8748db8d0b; no voters:
I20260619 16:39:44.230621 20912 leader_election.cc:288] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:44.230741 20922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:44.231032 20922 raft_consensus.cc:686] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [term 1 LEADER]: Becoming Leader. State: Replica: df366a638ad848bb8dec9c8748db8d0b, State: Running, Role: LEADER
I20260619 16:39:44.231304 20922 consensus_queue.cc:237] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df366a638ad848bb8dec9c8748db8d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } }
I20260619 16:39:44.231736 20912 sys_catalog.cc:564] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:44.233044 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df366a638ad848bb8dec9c8748db8d0b. Latest consensus state: current_term: 1 leader_uuid: "df366a638ad848bb8dec9c8748db8d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df366a638ad848bb8dec9c8748db8d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } } }
I20260619 16:39:44.233165 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:44.233132 20923 sys_catalog.cc:455] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df366a638ad848bb8dec9c8748db8d0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df366a638ad848bb8dec9c8748db8d0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36817 } } }
I20260619 16:39:44.233229 20923 sys_catalog.cc:458] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:44.234428 20931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:44.237555 20931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:44.242412 20931 catalog_manager.cc:1347] Generated new cluster ID: a0d5a31249fd45b6b1f1e8038d7fe946
I20260619 16:39:44.242496 20931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:44.267694 20931 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:44.267990 20931 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:44.278744 20931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df366a638ad848bb8dec9c8748db8d0b: Generated new TSK 0
I20260619 16:39:44.279378 20931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:44.282934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20844
2026-06-19T16:39:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:44 dist-test-slave-0f28 krb5kdc[20943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:44 dist-test-slave-0f28 krb5kdc[20943](info): set up 2 sockets
Jun 19 16:39:44 dist-test-slave-0f28 krb5kdc[20943](info): commencing operation
krb5kdc: starting...
W20260619 16:39:45.325429 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:45 dist-test-slave-0f28 krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:45.443967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:45.548821 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:45.549084 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:45.549154 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:45.553120 20956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:45.553205 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:45.553248 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:45.553285 20956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:45.553337 20956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:45.558396 20956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:45.559337 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:45.560276 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:45.568384 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:45.568408 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:45.568573 20956 server_base.cc:1034] running on GCE node
W20260619 16:39:45.568538 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:45.569154 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:45.570189 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:45.571372 20956 hybrid_clock.cc:648] HybridClock initialized: now 1781887185571341 us; error 40 us; skew 500 ppm
Jun 19 16:39:45 dist-test-slave-0f28 krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:45.578083 20956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:45.578985 20956 webserver.cc:460] Webserver started at http://127.0.0.1:35179/ using document root <none> and password file <none>
I20260619 16:39:45.579516 20956 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:45.579598 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:45.579823 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:45.581815 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a"
format_stamp: "Formatted at 2026-06-19 16:39:45 on dist-test-slave-0f28"
server_key: "b891d33b0688a9e2ee2a5e2e9dbebe30"
server_key_iv: "4e24f5404ed57e8c14349bdddd75beb9"
server_key_version: "encryptionkey@0"
I20260619 16:39:45.582348 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a"
format_stamp: "Formatted at 2026-06-19 16:39:45 on dist-test-slave-0f28"
server_key: "b891d33b0688a9e2ee2a5e2e9dbebe30"
server_key_iv: "4e24f5404ed57e8c14349bdddd75beb9"
server_key_version: "encryptionkey@0"
I20260619 16:39:45.585800 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 16:39:45.588279 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:45.589555 20956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:39:45.589679 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a"
format_stamp: "Formatted at 2026-06-19 16:39:45 on dist-test-slave-0f28"
server_key: "b891d33b0688a9e2ee2a5e2e9dbebe30"
server_key_iv: "4e24f5404ed57e8c14349bdddd75beb9"
server_key_version: "encryptionkey@0"
I20260619 16:39:45.589784 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:45.602546 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:45.605578 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:45.605759 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:45.612515 20956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40233
I20260619 16:39:45.612534 21023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40233 every 8 connection(s)
I20260619 16:39:45.613466 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:45.616015 21024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:45.619012 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 20956
I20260619 16:39:45.619145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:45.619407 392 external_mini_cluster.cc:1428] Setting key 92bbf9112ca283c8c4007404b794941a
I20260619 16:39:45.621711 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a: Bootstrap starting.
I20260619 16:39:45.624257 21024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:45.625078 21024 log.cc:824] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:45.627156 21024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a: No bootstrap required, opened a new log
Jun 19 16:39:45 dist-test-slave-0f28 krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:39:45.631507 21024 raft_consensus.cc:348] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40233 } }
I20260619 16:39:45.631770 21024 raft_consensus.cc:374] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:45.631845 21024 raft_consensus.cc:729] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff4fc0fc6a64a9e86523c8d6f9c0d4a, State: Initialized, Role: FOLLOWER
I20260619 16:39:45.632318 21024 consensus_queue.cc:260] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40233 } }
I20260619 16:39:45.632472 21024 raft_consensus.cc:388] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:45.632553 21024 raft_consensus.cc:482] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:45.632658 21024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:45.633764 21024 raft_consensus.cc:504] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40233 } }
I20260619 16:39:45.634110 21024 leader_election.cc:302] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff4fc0fc6a64a9e86523c8d6f9c0d4a; no voters:
I20260619 16:39:45.634431 21024 leader_election.cc:288] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:45.634562 21029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:45.634783 21029 raft_consensus.cc:686] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [term 1 LEADER]: Becoming Leader. State: Replica: eff4fc0fc6a64a9e86523c8d6f9c0d4a, State: Running, Role: LEADER
I20260619 16:39:45.634887 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:45.620721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58743 (local address 127.0.0.1:40233)
0619 16:39:45.621117 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:45.621135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:45.621175 (+ 40us) server_negotiation.cc:407] Connection header received
0619 16:39:45.622005 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:45.622027 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:45.622423 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:45.622781 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:45.623410 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:45.624884 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:45.626033 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:45.626687 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:45.626841 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:45.629566 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:45.629573 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:45.629592 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:45.629630 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:45.631577 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:45.632100 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:45.632106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:45.632111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:45.632240 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:45.632613 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:45.632616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:45.632618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:45.633108 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:45.633301 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:45.633672 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 16:39:45.633961 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":115,"threads_started":1}
I20260619 16:39:45.635098 21029 consensus_queue.cc:237] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40233 } }
I20260619 16:39:45.635558 21024 sys_catalog.cc:564] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:45.636626 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader eff4fc0fc6a64a9e86523c8d6f9c0d4a. Latest consensus state: current_term: 1 leader_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40233 } } }
I20260619 16:39:45.636750 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:45.637279 21031 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff4fc0fc6a64a9e86523c8d6f9c0d4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40233 } } }
I20260619 16:39:45.637392 21031 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:45.637921 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 16:39:45.640239 21044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:39:45.640321 21044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:39:45.640414 21038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:45.641373 21038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:45.647053 21038 catalog_manager.cc:1347] Generated new cluster ID: dd006c17addc473690bc7c22e94008d1
I20260619 16:39:45.647112 21038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:45.648389 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:45.639820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58771 (local address 127.0.0.1:40233)
0619 16:39:45.639914 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:45.639918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:45.640206 (+ 288us) server_negotiation.cc:407] Connection header received
0619 16:39:45.640315 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:45.640318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:45.640368 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:45.640447 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:45.641194 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:45.642019 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:45.643093 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:45.643332 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:45.643439 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:45.644704 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:45.644708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:45.644710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:45.644733 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:45.646659 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:45.647159 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:45.647163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:45.647165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:45.647252 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:45.647558 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:45.647561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:45.647563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:45.647765 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:45.647893 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:45.648121 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 16:39:45.648231 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:39:45.662465 21038 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:45.663645 21038 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:45.669981 21038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eff4fc0fc6a64a9e86523c8d6f9c0d4a: Generated new TSK 0
I20260619 16:39:45.670668 21038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:45.677138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 20956
2026-06-19T16:39:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:45 dist-test-slave-0f28 krb5kdc[21055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:45 dist-test-slave-0f28 krb5kdc[21055](info): set up 2 sockets
Jun 19 16:39:45 dist-test-slave-0f28 krb5kdc[21055](info): commencing operation
krb5kdc: starting...
W20260619 16:39:46.709584 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:46 dist-test-slave-0f28 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:46.828780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:46.943388 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:46.943682 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:46.943760 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:46.947690 21068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:46.947772 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:46.947810 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:46.947868 21068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:46.947906 21068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:46.953074 21068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:46.954102 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:46.955113 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:46.963474 21074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:46.963534 21076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:46.963537 21073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:46.963701 21068 server_base.cc:1034] running on GCE node
I20260619 16:39:46.964350 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:46.965461 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:46.966758 21068 hybrid_clock.cc:648] HybridClock initialized: now 1781887186966731 us; error 80 us; skew 500 ppm
Jun 19 16:39:46 dist-test-slave-0f28 krb5kdc[21055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:46.969519 21068 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:46.970409 21068 webserver.cc:460] Webserver started at http://127.0.0.1:45699/ using document root <none> and password file <none>
I20260619 16:39:46.970988 21068 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:46.971096 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:46.971345 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:46.973438 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "9dce18106e204e3983aec6b638a10204"
format_stamp: "Formatted at 2026-06-19 16:39:46 on dist-test-slave-0f28"
server_key: "8c5735c456e256e028674f1a15703531"
server_key_iv: "61e679d16975e9adb2d98970facb0356"
server_key_version: "encryptionkey@0"
I20260619 16:39:46.974001 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dce18106e204e3983aec6b638a10204"
format_stamp: "Formatted at 2026-06-19 16:39:46 on dist-test-slave-0f28"
server_key: "8c5735c456e256e028674f1a15703531"
server_key_iv: "61e679d16975e9adb2d98970facb0356"
server_key_version: "encryptionkey@0"
I20260619 16:39:46.977591 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:46.979841 21083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:46.980985 21068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:46.981117 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "9dce18106e204e3983aec6b638a10204"
format_stamp: "Formatted at 2026-06-19 16:39:46 on dist-test-slave-0f28"
server_key: "8c5735c456e256e028674f1a15703531"
server_key_iv: "61e679d16975e9adb2d98970facb0356"
server_key_version: "encryptionkey@0"
I20260619 16:39:46.981227 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:46.997937 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:47.000613 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:47.000808 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:47.007933 21068 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43367
I20260619 16:39:47.007958 21135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43367 every 8 connection(s)
I20260619 16:39:47.008862 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:47.011621 21136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:47.015205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21068
I20260619 16:39:47.015352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:47.015625 392 external_mini_cluster.cc:1428] Setting key a67d1fee7cc87cca024d65303f5a1f1b
I20260619 16:39:47.017424 21136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204: Bootstrap starting.
I20260619 16:39:47.019553 21136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:47.020272 21136 log.cc:824] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:47.022094 21136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204: No bootstrap required, opened a new log
I20260619 16:39:47.024395 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:47.016971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43533 (local address 127.0.0.1:43367)
0619 16:39:47.017432 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:47.017448 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:47.017486 (+ 38us) server_negotiation.cc:407] Connection header received
0619 16:39:47.018143 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:47.018160 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:47.018478 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:47.018792 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:47.019207 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:47.020210 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:47.021124 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:47.021639 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:47.021808 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:47.022606 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:47.022610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:47.022622 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:47.022994 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:47.023043 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:47.023064 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:47.023136 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:47.023296 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 16:39:47.023505 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260619 16:39:47.025302 21136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce18106e204e3983aec6b638a10204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260619 16:39:47.025585 21136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:47.025651 21136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dce18106e204e3983aec6b638a10204, State: Initialized, Role: FOLLOWER
I20260619 16:39:47.026085 21136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce18106e204e3983aec6b638a10204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260619 16:39:47.026278 21136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:47.026352 21136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:47.026456 21136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:47.027144 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:47.027498 21136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce18106e204e3983aec6b638a10204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260619 16:39:47.027855 21136 leader_election.cc:302] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dce18106e204e3983aec6b638a10204; no voters:
I20260619 16:39:47.028206 21136 leader_election.cc:288] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:47.028299 21141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:47.028529 21141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [term 1 LEADER]: Becoming Leader. State: Replica: 9dce18106e204e3983aec6b638a10204, State: Running, Role: LEADER
I20260619 16:39:47.028842 21141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce18106e204e3983aec6b638a10204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260619 16:39:47.029544 21136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:47.030000 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dce18106e204e3983aec6b638a10204. Latest consensus state: current_term: 1 leader_uuid: "9dce18106e204e3983aec6b638a10204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce18106e204e3983aec6b638a10204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } } }
I20260619 16:39:47.030788 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:47.030681 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dce18106e204e3983aec6b638a10204" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dce18106e204e3983aec6b638a10204" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } } }
I20260619 16:39:47.031056 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:47.031813 21155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:47.034310 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:47.029353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45051 (local address 127.0.0.1:43367)
0619 16:39:47.029448 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:47.029452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:47.029700 (+ 248us) server_negotiation.cc:407] Connection header received
0619 16:39:47.029829 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:47.029835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:47.029880 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:47.029956 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:47.030353 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:47.031143 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:47.032467 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:47.032766 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:47.032838 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:47.033845 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:47.033849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:47.033851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:47.033864 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:47.033890 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:47.033903 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:47.033946 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:47.034123 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:39:47.034185 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:39:47.035014 21155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:47.040515 21155 catalog_manager.cc:1347] Generated new cluster ID: 36ce2684233f42a79aa6d512abee5e4e
I20260619 16:39:47.040573 21155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:47.055553 21155 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:47.056736 21155 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:47.065752 21155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dce18106e204e3983aec6b638a10204: Generated new TSK 0
I20260619 16:39:47.066447 21155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:47.076268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21068
2026-06-19T16:39:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:47 dist-test-slave-0f28 krb5kdc[21167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:47 dist-test-slave-0f28 krb5kdc[21167](info): set up 2 sockets
Jun 19 16:39:47 dist-test-slave-0f28 krb5kdc[21167](info): commencing operation
krb5kdc: starting...
W20260619 16:39:48.121645 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:48 dist-test-slave-0f28 krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:48.244818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:48.356911 21180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:48.357164 21180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:48.357205 21180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:48.361340 21180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:48.361397 21180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:48.361418 21180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:48.361433 21180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:48.361483 21180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:48.366637 21180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:48.367617 21180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:48.368695 21180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:48.377163 21186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:48.377173 21188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:48.377215 21185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:48.377460 21180 server_base.cc:1034] running on GCE node
I20260619 16:39:48.378005 21180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:48.379007 21180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:48.380167 21180 hybrid_clock.cc:648] HybridClock initialized: now 1781887188380126 us; error 52 us; skew 500 ppm
Jun 19 16:39:48 dist-test-slave-0f28 krb5kdc[21167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:48.383033 21180 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:48.383999 21180 webserver.cc:460] Webserver started at http://127.0.0.1:44211/ using document root <none> and password file <none>
I20260619 16:39:48.384541 21180 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:48.384601 21180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:48.384832 21180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:48.386885 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "b918b05865ea4b29aa05fc5e0fc11e82"
format_stamp: "Formatted at 2026-06-19 16:39:48 on dist-test-slave-0f28"
server_key: "df8cefa906298403bf189b20016bf802"
server_key_iv: "ff8145efc9b0973b8df5b6713fad0223"
server_key_version: "encryptionkey@0"
I20260619 16:39:48.387434 21180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "b918b05865ea4b29aa05fc5e0fc11e82"
format_stamp: "Formatted at 2026-06-19 16:39:48 on dist-test-slave-0f28"
server_key: "df8cefa906298403bf189b20016bf802"
server_key_iv: "ff8145efc9b0973b8df5b6713fad0223"
server_key_version: "encryptionkey@0"
I20260619 16:39:48.391026 21180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:48.393409 21195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:48.394652 21180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:48.394773 21180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "b918b05865ea4b29aa05fc5e0fc11e82"
format_stamp: "Formatted at 2026-06-19 16:39:48 on dist-test-slave-0f28"
server_key: "df8cefa906298403bf189b20016bf802"
server_key_iv: "ff8145efc9b0973b8df5b6713fad0223"
server_key_version: "encryptionkey@0"
I20260619 16:39:48.394879 21180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:48.399307 21180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:48.402161 21180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:48.402343 21180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:48.409189 21180 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46617
I20260619 16:39:48.409224 21247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46617 every 8 connection(s)
I20260619 16:39:48.410105 21180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:48.410879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21180
I20260619 16:39:48.411020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:48.411269 392 external_mini_cluster.cc:1428] Setting key f5a6c5832c03ae299532b10a2b41d228
I20260619 16:39:48.412989 21248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:48.416230 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:48.412650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37821 (local address 127.0.0.1:46617)
0619 16:39:48.413034 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:48.413052 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:48.413115 (+ 63us) server_negotiation.cc:407] Connection header received
0619 16:39:48.413801 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:48.413819 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:48.414082 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:48.414355 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:48.415114 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:48.415118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:48.415129 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:48.415138 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:48.415183 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:48.415201 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:48.415302 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:48.415415 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 16:39:48.415576 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":137,"threads_started":1}
I20260619 16:39:48.418763 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:48.418901 21248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82: Bootstrap starting.
I20260619 16:39:48.421569 21248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:48.422446 21248 log.cc:824] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:48.422562 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:48.420905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56227 (local address 127.0.0.1:46617)
0619 16:39:48.421028 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:48.421047 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:48.421219 (+ 172us) server_negotiation.cc:407] Connection header received
0619 16:39:48.421365 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:48.421368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:48.421402 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:48.421469 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:48.422144 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:48.422147 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:48.422148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:48.422159 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:48.422181 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:48.422191 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:48.422253 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:48.422365 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:39:48.422393 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:39:48.424202 21248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82: No bootstrap required, opened a new log
I20260619 16:39:48.427456 21248 raft_consensus.cc:348] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260619 16:39:48.427675 21248 raft_consensus.cc:374] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:48.427744 21248 raft_consensus.cc:729] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b918b05865ea4b29aa05fc5e0fc11e82, State: Initialized, Role: FOLLOWER
I20260619 16:39:48.428118 21248 consensus_queue.cc:260] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260619 16:39:48.428256 21248 raft_consensus.cc:388] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:48.428334 21248 raft_consensus.cc:482] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:48.428426 21248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:48.429227 21248 raft_consensus.cc:504] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260619 16:39:48.429548 21248 leader_election.cc:302] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b918b05865ea4b29aa05fc5e0fc11e82; no voters:
I20260619 16:39:48.429814 21248 leader_election.cc:288] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:48.429916 21258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:48.430145 21258 raft_consensus.cc:686] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [term 1 LEADER]: Becoming Leader. State: Replica: b918b05865ea4b29aa05fc5e0fc11e82, State: Running, Role: LEADER
I20260619 16:39:48.430418 21258 consensus_queue.cc:237] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } }
I20260619 16:39:48.430917 21248 sys_catalog.cc:564] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:48.432034 21259 sys_catalog.cc:455] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } } }
I20260619 16:39:48.432080 21260 sys_catalog.cc:455] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b918b05865ea4b29aa05fc5e0fc11e82. Latest consensus state: current_term: 1 leader_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b918b05865ea4b29aa05fc5e0fc11e82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46617 } } }
I20260619 16:39:48.432219 21259 sys_catalog.cc:458] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:48.432235 21260 sys_catalog.cc:458] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:48.433164 21268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:48.435899 21268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:48.440479 21268 catalog_manager.cc:1347] Generated new cluster ID: a864e038f0c44dc2adcd4a8dc3834a42
I20260619 16:39:48.440538 21268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:48.472450 21268 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:48.472810 21268 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:48.481711 21268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b918b05865ea4b29aa05fc5e0fc11e82: Generated new TSK 0
I20260619 16:39:48.482542 21268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:48.497361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21180
2026-06-19T16:39:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:48 dist-test-slave-0f28 krb5kdc[21279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:48 dist-test-slave-0f28 krb5kdc[21279](info): set up 2 sockets
Jun 19 16:39:48 dist-test-slave-0f28 krb5kdc[21279](info): commencing operation
krb5kdc: starting...
W20260619 16:39:49.541584 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:49 dist-test-slave-0f28 krb5kdc[21279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:49.664392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:49.769238 21292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:49.769490 21292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:49.769528 21292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:49.773353 21292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:49.773420 21292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:49.773442 21292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:49.773458 21292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:49.773478 21292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:49.778402 21292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:49.779227 21292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:49.780082 21292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:49.788117 21297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:49.788131 21298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:49.788125 21300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:49.788208 21292 server_base.cc:1034] running on GCE node
I20260619 16:39:49.788830 21292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:49.789947 21292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:49.791105 21292 hybrid_clock.cc:648] HybridClock initialized: now 1781887189791078 us; error 39 us; skew 500 ppm
Jun 19 16:39:49 dist-test-slave-0f28 krb5kdc[21279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:49.793671 21292 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:49.794600 21292 webserver.cc:460] Webserver started at http://127.0.0.1:41853/ using document root <none> and password file <none>
I20260619 16:39:49.795127 21292 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:49.795212 21292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:49.795424 21292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:49.797286 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "1230565e9cf146b992c118f7fc85db14"
format_stamp: "Formatted at 2026-06-19 16:39:49 on dist-test-slave-0f28"
server_key: "892fb661bf4f53842731113519c1a25a"
server_key_iv: "c14f559b1a3440baa65855c83f0997b2"
server_key_version: "encryptionkey@0"
I20260619 16:39:49.797926 21292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "1230565e9cf146b992c118f7fc85db14"
format_stamp: "Formatted at 2026-06-19 16:39:49 on dist-test-slave-0f28"
server_key: "892fb661bf4f53842731113519c1a25a"
server_key_iv: "c14f559b1a3440baa65855c83f0997b2"
server_key_version: "encryptionkey@0"
I20260619 16:39:49.801371 21292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:39:49.803561 21307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:49.804689 21292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:49.804819 21292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "1230565e9cf146b992c118f7fc85db14"
format_stamp: "Formatted at 2026-06-19 16:39:49 on dist-test-slave-0f28"
server_key: "892fb661bf4f53842731113519c1a25a"
server_key_iv: "c14f559b1a3440baa65855c83f0997b2"
server_key_version: "encryptionkey@0"
I20260619 16:39:49.804932 21292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:49.818457 21292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:49.821074 21292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:49.821254 21292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:49.827792 21292 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35333
I20260619 16:39:49.827821 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35333 every 8 connection(s)
I20260619 16:39:49.828642 21292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:49.829653 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21292
I20260619 16:39:49.829792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:49.830106 392 external_mini_cluster.cc:1428] Setting key a3059c4b956579ae0d1b3b1f33eb8870
I20260619 16:39:49.831646 21360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:49.836800 21360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14: Bootstrap starting.
Jun 19 16:39:49 dist-test-slave-0f28 krb5kdc[21279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:39:49.838920 21360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:49.839751 21360 log.cc:824] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:49.841526 21360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14: No bootstrap required, opened a new log
I20260619 16:39:49.843258 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:49.831349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45413 (local address 127.0.0.1:35333)
0619 16:39:49.831797 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:49.831812 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:49.831844 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:39:49.832460 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:49.832475 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:49.832783 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:49.833103 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:49.833897 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:49.834828 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:49.835668 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:49.836116 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:49.836276 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:49.838707 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:49.838712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:49.838724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:49.838748 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:49.840558 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:49.841029 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:49.841035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:49.841040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:49.841146 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:49.841515 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:49.841518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:49.841520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:49.841944 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:49.842159 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:49.842427 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 16:39:49.842625 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260619 16:39:49.845144 21360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1230565e9cf146b992c118f7fc85db14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35333 } }
I20260619 16:39:49.845398 21360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:49.845475 21360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1230565e9cf146b992c118f7fc85db14, State: Initialized, Role: FOLLOWER
I20260619 16:39:49.845872 21360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1230565e9cf146b992c118f7fc85db14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35333 } }
I20260619 16:39:49.846050 21360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:49.846077 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:49.846120 21360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:49.846230 21360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:49.847033 21360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1230565e9cf146b992c118f7fc85db14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35333 } }
I20260619 16:39:49.847422 21360 leader_election.cc:302] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1230565e9cf146b992c118f7fc85db14; no voters:
I20260619 16:39:49.847713 21360 leader_election.cc:288] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:49.847779 21369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:49.847985 21369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [term 1 LEADER]: Becoming Leader. State: Replica: 1230565e9cf146b992c118f7fc85db14, State: Running, Role: LEADER
I20260619 16:39:49.848248 21369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1230565e9cf146b992c118f7fc85db14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35333 } }
I20260619 16:39:49.848881 21360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:49.849339 21369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1230565e9cf146b992c118f7fc85db14. Latest consensus state: current_term: 1 leader_uuid: "1230565e9cf146b992c118f7fc85db14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1230565e9cf146b992c118f7fc85db14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35333 } } }
I20260619 16:39:49.849498 21369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:49.849745 21369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1230565e9cf146b992c118f7fc85db14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1230565e9cf146b992c118f7fc85db14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35333 } } }
I20260619 16:39:49.849887 21369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:49.850265 21376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:49.854934 21376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:49.856729 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:49.847971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:53031 (local address 127.0.0.1:35333)
0619 16:39:49.848064 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:49.848069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:49.848402 (+ 333us) server_negotiation.cc:407] Connection header received
0619 16:39:49.848568 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:49.848571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:49.848610 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:49.848692 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:49.849438 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:49.850178 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:49.851272 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:49.851524 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:49.851631 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:49.853145 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:49.853148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:49.853151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:49.853175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:49.854878 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:49.855407 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:49.855411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:49.855420 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:49.855503 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:49.855863 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:49.855866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:49.855868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:49.856055 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:49.856190 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:49.856496 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 16:39:49.856584 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 16:39:49.859972 21376 catalog_manager.cc:1347] Generated new cluster ID: 0fcc821fbcfe46b6a08914694a6e54a9
I20260619 16:39:49.860031 21376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:49.877386 21376 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:49.878319 21376 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:49.885855 21376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1230565e9cf146b992c118f7fc85db14: Generated new TSK 0
I20260619 16:39:49.886457 21376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:49.892217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21292
W20260619 16:39:49.896821 21277 connection.cc:535] client connection to 127.0.0.1:35333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35333): Connection reset by peer (error 104)
2026-06-19T16:39:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:49 dist-test-slave-0f28 krb5kdc[21391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:49 dist-test-slave-0f28 krb5kdc[21391](info): set up 2 sockets
Jun 19 16:39:49 dist-test-slave-0f28 krb5kdc[21391](info): commencing operation
krb5kdc: starting...
W20260619 16:39:50.937529 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:51 dist-test-slave-0f28 krb5kdc[21391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:51.055809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:51.162698 21404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:51.162933 21404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:51.162972 21404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:51.166819 21404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:51.166872 21404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:51.166890 21404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:51.166913 21404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:51.166927 21404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:51.171990 21404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:51.172855 21404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:51.173789 21404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:51.181687 21410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:51.181754 21404 server_base.cc:1034] running on GCE node
W20260619 16:39:51.181701 21412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:51.181703 21409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:51.182410 21404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:51.183444 21404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:51.184592 21404 hybrid_clock.cc:648] HybridClock initialized: now 1781887191184569 us; error 37 us; skew 500 ppm
Jun 19 16:39:51 dist-test-slave-0f28 krb5kdc[21391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:51.187165 21404 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:51.187994 21404 webserver.cc:460] Webserver started at http://127.0.0.1:34673/ using document root <none> and password file <none>
I20260619 16:39:51.188504 21404 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:51.188585 21404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:51.188792 21404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:51.190712 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "735a5a0ceb1e44b1881326bef7b35423"
format_stamp: "Formatted at 2026-06-19 16:39:51 on dist-test-slave-0f28"
server_key: "d994994da83c5d80595726377eb2b53c"
server_key_iv: "e11db0df605bcdd7b13d3024e621313f"
server_key_version: "encryptionkey@0"
I20260619 16:39:51.191294 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "735a5a0ceb1e44b1881326bef7b35423"
format_stamp: "Formatted at 2026-06-19 16:39:51 on dist-test-slave-0f28"
server_key: "d994994da83c5d80595726377eb2b53c"
server_key_iv: "e11db0df605bcdd7b13d3024e621313f"
server_key_version: "encryptionkey@0"
I20260619 16:39:51.194731 21404 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:39:51.196885 21419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:51.198069 21404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:51.198194 21404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "735a5a0ceb1e44b1881326bef7b35423"
format_stamp: "Formatted at 2026-06-19 16:39:51 on dist-test-slave-0f28"
server_key: "d994994da83c5d80595726377eb2b53c"
server_key_iv: "e11db0df605bcdd7b13d3024e621313f"
server_key_version: "encryptionkey@0"
I20260619 16:39:51.198295 21404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:51.226264 21404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:51.228834 21404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:51.229024 21404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:51.235677 21404 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37889
I20260619 16:39:51.235702 21471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37889 every 8 connection(s)
I20260619 16:39:51.236541 21404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:51.239097 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:51.241569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21404
I20260619 16:39:51.241698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:51.241977 392 external_mini_cluster.cc:1428] Setting key f3beb367821677aa737d0c1d54989f16
I20260619 16:39:51.245078 21472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423: Bootstrap starting.
I20260619 16:39:51.247589 21472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:51.248363 21472 log.cc:824] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:51.250366 21472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423: No bootstrap required, opened a new log
I20260619 16:39:51.251699 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:51.243243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:60263 (local address 127.0.0.1:37889)
0619 16:39:51.243658 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:51.243678 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:51.243719 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:39:51.244517 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:51.244539 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:51.244950 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:51.245294 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:51.245704 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:51.247135 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:51.248144 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:51.248815 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:51.248937 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:51.249826 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:51.249833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:51.249847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:51.250311 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:51.250374 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:51.250409 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:51.250534 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:51.250711 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 16:39:51.250839 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":112,"threads_started":1}
I20260619 16:39:51.253994 21472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735a5a0ceb1e44b1881326bef7b35423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260619 16:39:51.254232 21472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:51.254344 21472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 735a5a0ceb1e44b1881326bef7b35423, State: Initialized, Role: FOLLOWER
I20260619 16:39:51.254825 21472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735a5a0ceb1e44b1881326bef7b35423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260619 16:39:51.254915 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:51.255002 21472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:51.255069 21472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:51.255160 21472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:51.255983 21472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735a5a0ceb1e44b1881326bef7b35423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260619 16:39:51.256349 21472 leader_election.cc:302] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 735a5a0ceb1e44b1881326bef7b35423; no voters:
I20260619 16:39:51.256651 21472 leader_election.cc:288] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:51.256752 21481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:51.257050 21481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [term 1 LEADER]: Becoming Leader. State: Replica: 735a5a0ceb1e44b1881326bef7b35423, State: Running, Role: LEADER
I20260619 16:39:51.257444 21481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735a5a0ceb1e44b1881326bef7b35423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } }
I20260619 16:39:51.257928 21472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:51.259056 21483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "735a5a0ceb1e44b1881326bef7b35423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735a5a0ceb1e44b1881326bef7b35423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } } }
I20260619 16:39:51.259107 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 735a5a0ceb1e44b1881326bef7b35423. Latest consensus state: current_term: 1 leader_uuid: "735a5a0ceb1e44b1881326bef7b35423" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "735a5a0ceb1e44b1881326bef7b35423" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37889 } } }
I20260619 16:39:51.259239 21483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:51.259243 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:51.259662 21486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:51.262468 21486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:51.262670 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:51.256872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45499 (local address 127.0.0.1:37889)
0619 16:39:51.256983 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:51.256987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:51.257476 (+ 489us) server_negotiation.cc:407] Connection header received
0619 16:39:51.257778 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:51.257781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:51.257824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:51.257870 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:51.258338 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:51.259196 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:51.260396 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:51.260771 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:51.260855 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:51.262030 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:51.262034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:51.262037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:51.262050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:51.262077 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:51.262090 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:51.262148 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:51.262496 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 16:39:51.262529 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260619 16:39:51.267235 21486 catalog_manager.cc:1347] Generated new cluster ID: bf5db6f5ba2c4555b8be20785bd17999
I20260619 16:39:51.267294 21486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:51.276151 21486 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:51.276968 21486 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:51.281347 21486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 735a5a0ceb1e44b1881326bef7b35423: Generated new TSK 0
I20260619 16:39:51.282040 21486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:51.290454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21404
W20260619 16:39:51.295409 21389 connection.cc:535] client connection to 127.0.0.1:37889 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37889): Connection reset by peer (error 104)
2026-06-19T16:39:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:51 dist-test-slave-0f28 krb5kdc[21503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:51 dist-test-slave-0f28 krb5kdc[21503](info): set up 2 sockets
Jun 19 16:39:51 dist-test-slave-0f28 krb5kdc[21503](info): commencing operation
krb5kdc: starting...
W20260619 16:39:52.341555 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:52 dist-test-slave-0f28 krb5kdc[21503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:52.459818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37309
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:52.566133 21516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:52.566413 21516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:52.566498 21516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:52.570406 21516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:52.570492 21516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:52.570533 21516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:52.570567 21516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:52.570603 21516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:52.575794 21516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:52.576750 21516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:52.577751 21516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:52.585772 21522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:52.585785 21524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:52.585815 21521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:52.585865 21516 server_base.cc:1034] running on GCE node
I20260619 16:39:52.586572 21516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:52.587582 21516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:52.588752 21516 hybrid_clock.cc:648] HybridClock initialized: now 1781887192588731 us; error 30 us; skew 500 ppm
Jun 19 16:39:52 dist-test-slave-0f28 krb5kdc[21503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:52.591519 21516 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:52.592417 21516 webserver.cc:460] Webserver started at http://127.0.0.1:42299/ using document root <none> and password file <none>
I20260619 16:39:52.593009 21516 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:52.593106 21516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:52.593410 21516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:52.595554 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "994a7054cc9d46ccba8a37dbde6093d8"
format_stamp: "Formatted at 2026-06-19 16:39:52 on dist-test-slave-0f28"
server_key: "39fc72f3e156992d653215aa6103dc05"
server_key_iv: "0069e8f513f876c0da2858eaf2ee3ae5"
server_key_version: "encryptionkey@0"
I20260619 16:39:52.596066 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "994a7054cc9d46ccba8a37dbde6093d8"
format_stamp: "Formatted at 2026-06-19 16:39:52 on dist-test-slave-0f28"
server_key: "39fc72f3e156992d653215aa6103dc05"
server_key_iv: "0069e8f513f876c0da2858eaf2ee3ae5"
server_key_version: "encryptionkey@0"
I20260619 16:39:52.599581 21516 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:52.601859 21531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:52.603029 21516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:52.603123 21516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "994a7054cc9d46ccba8a37dbde6093d8"
format_stamp: "Formatted at 2026-06-19 16:39:52 on dist-test-slave-0f28"
server_key: "39fc72f3e156992d653215aa6103dc05"
server_key_iv: "0069e8f513f876c0da2858eaf2ee3ae5"
server_key_version: "encryptionkey@0"
I20260619 16:39:52.603210 21516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:52.607592 21516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:52.610750 21516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:52.610898 21516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:52.618007 21516 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37309
I20260619 16:39:52.618037 21583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37309 every 8 connection(s)
I20260619 16:39:52.618870 21516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:52.621426 21584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:52.625089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21516
I20260619 16:39:52.625231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:52.625522 392 external_mini_cluster.cc:1428] Setting key 13d658d9cb7cb3074f183f804b29f62f
I20260619 16:39:52.627025 21584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8: Bootstrap starting.
I20260619 16:39:52.629240 21584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:52.630079 21584 log.cc:824] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:52.630594 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:52.626862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:51543 (local address 127.0.0.1:37309)
0619 16:39:52.627311 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:52.627326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:52.627359 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:39:52.627968 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:52.627982 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:52.628249 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:52.628584 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:52.629378 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:52.629383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:52.629399 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:52.629411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:52.629475 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:52.629496 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:52.629605 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:52.629717 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 16:39:52.629877 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260619 16:39:52.632019 21584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8: No bootstrap required, opened a new log
I20260619 16:39:52.633358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:52.636047 21584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994a7054cc9d46ccba8a37dbde6093d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260619 16:39:52.636327 21584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:52.636399 21584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994a7054cc9d46ccba8a37dbde6093d8, State: Initialized, Role: FOLLOWER
I20260619 16:39:52.636907 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:52.635279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:57585 (local address 127.0.0.1:37309)
0619 16:39:52.635401 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:52.635405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:52.635570 (+ 165us) server_negotiation.cc:407] Connection header received
0619 16:39:52.635680 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:52.635682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:52.635708 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:52.635769 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:52.636479 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:52.636482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:52.636484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:52.636495 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:52.636522 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:52.636534 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:52.636616 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:52.636755 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 16:39:52.636798 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 16:39:52.636881 21584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994a7054cc9d46ccba8a37dbde6093d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260619 16:39:52.637027 21584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:52.637096 21584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:52.637194 21584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:52.638144 21584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994a7054cc9d46ccba8a37dbde6093d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260619 16:39:52.638427 21584 leader_election.cc:302] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 994a7054cc9d46ccba8a37dbde6093d8; no voters:
I20260619 16:39:52.638648 21584 leader_election.cc:288] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:52.638777 21594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:52.638985 21594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [term 1 LEADER]: Becoming Leader. State: Replica: 994a7054cc9d46ccba8a37dbde6093d8, State: Running, Role: LEADER
I20260619 16:39:52.639328 21594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994a7054cc9d46ccba8a37dbde6093d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } }
I20260619 16:39:52.639803 21584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:52.641070 21595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "994a7054cc9d46ccba8a37dbde6093d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994a7054cc9d46ccba8a37dbde6093d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } } }
I20260619 16:39:52.641052 21596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 994a7054cc9d46ccba8a37dbde6093d8. Latest consensus state: current_term: 1 leader_uuid: "994a7054cc9d46ccba8a37dbde6093d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994a7054cc9d46ccba8a37dbde6093d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37309 } } }
I20260619 16:39:52.641220 21595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:52.641222 21596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:52.642117 21604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:52.644743 21604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:52.649857 21604 catalog_manager.cc:1347] Generated new cluster ID: 91c42cc6194647a59433d6330aed04e0
I20260619 16:39:52.649916 21604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:52.659482 21604 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:52.659752 21604 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:52.665807 21604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 994a7054cc9d46ccba8a37dbde6093d8: Generated new TSK 0
I20260619 16:39:52.666471 21604 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:52.675604 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21516
2026-06-19T16:39:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:52 dist-test-slave-0f28 krb5kdc[21615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:52 dist-test-slave-0f28 krb5kdc[21615](info): set up 2 sockets
Jun 19 16:39:52 dist-test-slave-0f28 krb5kdc[21615](info): commencing operation
krb5kdc: starting...
W20260619 16:39:53.717600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:53 dist-test-slave-0f28 krb5kdc[21615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:53.838517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33709
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:53.952562 21628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:53.952847 21628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:53.952936 21628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:53.956897 21628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:53.956986 21628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:53.957033 21628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:53.957089 21628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:53.957125 21628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:53.962363 21628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33709
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:53.963377 21628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:53.964354 21628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:53.972836 21636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:53.972815 21633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:53.972890 21628 server_base.cc:1034] running on GCE node
W20260619 16:39:53.972831 21634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:53.973599 21628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:53.974690 21628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:53.975852 21628 hybrid_clock.cc:648] HybridClock initialized: now 1781887193975838 us; error 25 us; skew 500 ppm
Jun 19 16:39:53 dist-test-slave-0f28 krb5kdc[21615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:53.978804 21628 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:53.979770 21628 webserver.cc:460] Webserver started at http://127.0.0.1:40545/ using document root <none> and password file <none>
I20260619 16:39:53.980350 21628 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:53.980443 21628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:53.980676 21628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:53.982851 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17"
format_stamp: "Formatted at 2026-06-19 16:39:53 on dist-test-slave-0f28"
server_key: "d22ccc007b34238e57546eb07b9e1140"
server_key_iv: "82d3514704caa5a54b0a7e98594735c4"
server_key_version: "encryptionkey@0"
I20260619 16:39:53.983430 21628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17"
format_stamp: "Formatted at 2026-06-19 16:39:53 on dist-test-slave-0f28"
server_key: "d22ccc007b34238e57546eb07b9e1140"
server_key_iv: "82d3514704caa5a54b0a7e98594735c4"
server_key_version: "encryptionkey@0"
I20260619 16:39:53.987228 21628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 16:39:53.989600 21643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:53.990832 21628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:53.990963 21628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17"
format_stamp: "Formatted at 2026-06-19 16:39:53 on dist-test-slave-0f28"
server_key: "d22ccc007b34238e57546eb07b9e1140"
server_key_iv: "82d3514704caa5a54b0a7e98594735c4"
server_key_version: "encryptionkey@0"
I20260619 16:39:53.991067 21628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:54.028031 21628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:54.031451 21628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:54.031670 21628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:54.039361 21628 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33709
I20260619 16:39:54.039382 21695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33709 every 8 connection(s)
I20260619 16:39:54.040427 21628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:54.043035 21696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:54.044682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21628
I20260619 16:39:54.044834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:54.045120 392 external_mini_cluster.cc:1428] Setting key f806e62a511e09a47d7e449a51b43b6a
I20260619 16:39:54.049386 21696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17: Bootstrap starting.
I20260619 16:39:54.051857 21696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:54.052776 21696 log.cc:824] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17: Log is configured to *not* fsync() on all Append() calls
Jun 19 16:39:54 dist-test-slave-0f28 krb5kdc[21615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 16:39:54.054816 21696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17: No bootstrap required, opened a new log
I20260619 16:39:54.059581 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:54.046409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:56337 (local address 127.0.0.1:33709)
0619 16:39:54.046787 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:54.046803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:54.046835 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:39:54.047445 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:54.047460 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:54.047774 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:54.048046 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:54.048925 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:54.049989 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:54.051138 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:54.051584 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:54.051696 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:54.054538 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:54.054544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:54.054560 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:54.054590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:54.056496 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:54.056986 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:54.056993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:54.057001 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:54.057158 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:54.057546 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:54.057549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:54.057551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:54.058025 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:54.058247 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:54.058442 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 16:39:54.058650 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":120,"threads_started":1}
I20260619 16:39:54.059499 21696 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 16:39:54.059739 21696 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:54.059800 21696 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c1e13ee8394028a4aaf21e4fdbfb17, State: Initialized, Role: FOLLOWER
I20260619 16:39:54.060245 21696 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 16:39:54.060403 21696 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:54.060506 21696 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:54.060662 21696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:54.061693 21696 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 16:39:54.062019 21696 leader_election.cc:302] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c1e13ee8394028a4aaf21e4fdbfb17; no voters:
I20260619 16:39:54.062373 21696 leader_election.cc:288] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:54.062450 21701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:54.062532 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:54.062660 21701 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [term 1 LEADER]: Becoming Leader. State: Replica: e3c1e13ee8394028a4aaf21e4fdbfb17, State: Running, Role: LEADER
I20260619 16:39:54.063002 21701 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } }
I20260619 16:39:54.063627 21696 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:54.064821 21702 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } } }
I20260619 16:39:54.064971 21702 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:54.064941 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c1e13ee8394028a4aaf21e4fdbfb17. Latest consensus state: current_term: 1 leader_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c1e13ee8394028a4aaf21e4fdbfb17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33709 } } }
I20260619 16:39:54.065043 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17 [sys.catalog]: This master's current role is: LEADER
W20260619 16:39:54.068351 21721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:39:54.068460 21721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:39:54.068568 21715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:54.069248 21715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:54.074571 21699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:54.064816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:53125 (local address 127.0.0.1:33709)
0619 16:39:54.065219 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:54.065226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:54.065538 (+ 312us) server_negotiation.cc:407] Connection header received
0619 16:39:54.065733 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:54.065736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:54.065787 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:54.065847 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:54.066801 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:54.067477 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:54.068638 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:54.068958 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:54.069030 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:54.070888 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:54.070892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:54.070894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:54.070924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:54.072656 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:54.073273 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:54.073275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:54.073276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:54.073408 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:54.073761 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:54.073763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:54.073764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:54.073924 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:54.074031 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:54.074336 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 16:39:54.074465 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260619 16:39:54.078729 21715 catalog_manager.cc:1347] Generated new cluster ID: b744c2276cc54377ab42a325b10adec4
I20260619 16:39:54.078786 21715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:54.094183 21715 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:54.095232 21715 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:54.107859 21715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c1e13ee8394028a4aaf21e4fdbfb17: Generated new TSK 0
I20260619 16:39:54.108556 21715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:54.118001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21628
W20260619 16:39:54.123322 21613 connection.cc:535] client connection to 127.0.0.1:33709 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33709): Connection reset by peer (error 104)
2026-06-19T16:39:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:54 dist-test-slave-0f28 krb5kdc[21727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:54 dist-test-slave-0f28 krb5kdc[21727](info): set up 2 sockets
Jun 19 16:39:54 dist-test-slave-0f28 krb5kdc[21727](info): commencing operation
krb5kdc: starting...
W20260619 16:39:55.170424 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:55 dist-test-slave-0f28 krb5kdc[21727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:55.292768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:55.398885 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:55.399147 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:55.399216 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:55.403138 21740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:55.403216 21740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:55.403342 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:55.403403 21740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:55.403438 21740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:55.408484 21740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:55.409435 21740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:55.410459 21740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:55.418690 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:55.418701 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:55.418759 21740 server_base.cc:1034] running on GCE node
W20260619 16:39:55.418701 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:55.419409 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:55.420403 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:55.421579 21740 hybrid_clock.cc:648] HybridClock initialized: now 1781887195421567 us; error 25 us; skew 500 ppm
Jun 19 16:39:55 dist-test-slave-0f28 krb5kdc[21727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:55.424160 21740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:55.425006 21740 webserver.cc:460] Webserver started at http://127.0.0.1:34891/ using document root <none> and password file <none>
I20260619 16:39:55.425556 21740 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:55.425616 21740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:55.425769 21740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:55.427610 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa"
format_stamp: "Formatted at 2026-06-19 16:39:55 on dist-test-slave-0f28"
server_key: "f918f688dacfdbd99947623470715cdb"
server_key_iv: "1a1143ee92ee1538d9caff1752e7abaa"
server_key_version: "encryptionkey@0"
I20260619 16:39:55.428084 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa"
format_stamp: "Formatted at 2026-06-19 16:39:55 on dist-test-slave-0f28"
server_key: "f918f688dacfdbd99947623470715cdb"
server_key_iv: "1a1143ee92ee1538d9caff1752e7abaa"
server_key_version: "encryptionkey@0"
I20260619 16:39:55.431559 21740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:39:55.433887 21755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:55.435016 21740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:39:55.435109 21740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa"
format_stamp: "Formatted at 2026-06-19 16:39:55 on dist-test-slave-0f28"
server_key: "f918f688dacfdbd99947623470715cdb"
server_key_iv: "1a1143ee92ee1538d9caff1752e7abaa"
server_key_version: "encryptionkey@0"
I20260619 16:39:55.435227 21740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:55.459528 21740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:55.462633 21740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:55.462777 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:55.469825 21740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35497
I20260619 16:39:55.469849 21807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35497 every 8 connection(s)
I20260619 16:39:55.470715 21740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:55.473241 21808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:55.478578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21740
I20260619 16:39:55.478648 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa: Bootstrap starting.
I20260619 16:39:55.478708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:55.478981 392 external_mini_cluster.cc:1428] Setting key d332dca2f0e5f1f3b36d481e5a5b76f1
I20260619 16:39:55.481115 21808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:55.481865 21808 log.cc:824] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:55.483593 21808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa: No bootstrap required, opened a new log
I20260619 16:39:55.487125 21808 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260619 16:39:55.487347 21808 raft_consensus.cc:374] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:55.487383 21808 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebb5a11929e4ddba4dec6b9d5f6d9fa, State: Initialized, Role: FOLLOWER
I20260619 16:39:55.487772 21808 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260619 16:39:55.487890 21808 raft_consensus.cc:388] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:55.487972 21808 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:55.488075 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:55.480340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45751 (local address 127.0.0.1:35497)
0619 16:39:55.480765 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:55.480779 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:55.480812 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:39:55.481576 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:55.481595 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:55.481919 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:55.482316 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:55.482712 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:55.483681 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:55.484583 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:55.485041 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:55.485223 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:55.486149 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:55.486153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:55.486164 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:55.486569 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:55.486619 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:55.486641 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:55.486783 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:55.486972 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 16:39:55.487102 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260619 16:39:55.488091 21808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:55.488945 21808 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260619 16:39:55.489252 21808 leader_election.cc:302] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eebb5a11929e4ddba4dec6b9d5f6d9fa; no voters:
I20260619 16:39:55.489576 21808 leader_election.cc:288] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:55.489660 21813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:55.489853 21813 raft_consensus.cc:686] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [term 1 LEADER]: Becoming Leader. State: Replica: eebb5a11929e4ddba4dec6b9d5f6d9fa, State: Running, Role: LEADER
I20260619 16:39:55.490167 21813 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260619 16:39:55.490760 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:55.490763 21808 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:55.491909 21815 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } } }
I20260619 16:39:55.491910 21814 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebb5a11929e4ddba4dec6b9d5f6d9fa. Latest consensus state: current_term: 1 leader_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebb5a11929e4ddba4dec6b9d5f6d9fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } } }
I20260619 16:39:55.492228 21815 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:55.492236 21814 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa [sys.catalog]: This master's current role is: LEADER
W20260619 16:39:55.495478 21832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:39:55.495559 21832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:39:55.495616 21828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:55.496414 21828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:55.497642 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:55.493803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:45363 (local address 127.0.0.1:35497)
0619 16:39:55.493912 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:55.493917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:55.494148 (+ 231us) server_negotiation.cc:407] Connection header received
0619 16:39:55.494281 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:55.494284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:55.494322 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:55.494405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:55.494643 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:55.495159 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:55.496062 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:55.496295 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:55.496400 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:55.497191 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:55.497195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:55.497197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:55.497209 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:55.497238 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:55.497248 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:55.497368 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:55.497482 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 16:39:55.497513 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 16:39:55.501669 21828 catalog_manager.cc:1347] Generated new cluster ID: 53350d2d352b43b589723bd630f3c543
I20260619 16:39:55.501734 21828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:55.511410 21828 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:55.512225 21828 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:55.520123 21828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eebb5a11929e4ddba4dec6b9d5f6d9fa: Generated new TSK 0
I20260619 16:39:55.520709 21828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:55.526904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21740
W20260619 16:39:55.532007 21725 connection.cc:535] client connection to 127.0.0.1:35497 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35497): Connection reset by peer (error 104)
2026-06-19T16:39:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:55 dist-test-slave-0f28 krb5kdc[21839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:55 dist-test-slave-0f28 krb5kdc[21839](info): set up 2 sockets
Jun 19 16:39:55 dist-test-slave-0f28 krb5kdc[21839](info): commencing operation
krb5kdc: starting...
W20260619 16:39:56.561580 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:56 dist-test-slave-0f28 krb5kdc[21839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 16:39:56.680727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:56.786463 21852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:56.786767 21852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:56.786825 21852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:56.790829 21852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:56.790879 21852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:56.790899 21852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:56.790915 21852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:56.790930 21852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:56.795883 21852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:56.796702 21852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:56.797611 21852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:56.805581 21858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:56.805636 21852 server_base.cc:1034] running on GCE node
W20260619 16:39:56.805567 21857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:56.805581 21860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:56.806239 21852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:56.807320 21852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:56.808530 21852 hybrid_clock.cc:648] HybridClock initialized: now 1781887196808512 us; error 30 us; skew 500 ppm
Jun 19 16:39:56 dist-test-slave-0f28 krb5kdc[21839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:56.815388 21852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 16:39:56.816439 21852 webserver.cc:460] Webserver started at http://127.0.0.1:42905/ using document root <none> and password file <none>
I20260619 16:39:56.817050 21852 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:56.817152 21852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:56.817464 21852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:56.819657 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "30383773a2fb4b15b1f41d9e0832dee4"
format_stamp: "Formatted at 2026-06-19 16:39:56 on dist-test-slave-0f28"
server_key: "c1bb29bb622753521f0699b28ad088aa"
server_key_iv: "6dacb7fa9eeab867d4c439c736096e92"
server_key_version: "encryptionkey@0"
I20260619 16:39:56.820277 21852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "30383773a2fb4b15b1f41d9e0832dee4"
format_stamp: "Formatted at 2026-06-19 16:39:56 on dist-test-slave-0f28"
server_key: "c1bb29bb622753521f0699b28ad088aa"
server_key_iv: "6dacb7fa9eeab867d4c439c736096e92"
server_key_version: "encryptionkey@0"
I20260619 16:39:56.824168 21852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 16:39:56.826668 21867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:56.827877 21852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:56.828035 21852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "30383773a2fb4b15b1f41d9e0832dee4"
format_stamp: "Formatted at 2026-06-19 16:39:56 on dist-test-slave-0f28"
server_key: "c1bb29bb622753521f0699b28ad088aa"
server_key_iv: "6dacb7fa9eeab867d4c439c736096e92"
server_key_version: "encryptionkey@0"
I20260619 16:39:56.828150 21852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:56.832517 21852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:56.835389 21852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:56.835572 21852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:56.842593 21852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33305
I20260619 16:39:56.842617 21919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33305 every 8 connection(s)
I20260619 16:39:56.843437 21852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:56.846094 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21852
I20260619 16:39:56.846220 21920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:56.846246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:56.846553 392 external_mini_cluster.cc:1428] Setting key eb910391480d7978352cb398a0faa280
I20260619 16:39:56.851778 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:56.847882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:47763 (local address 127.0.0.1:33305)
0619 16:39:56.848319 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:56.848333 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:56.848366 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:39:56.849228 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:56.849261 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:56.849598 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:56.849920 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:56.850679 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:56.850683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:56.850694 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:56.850705 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:56.850759 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:56.850777 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:56.850921 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:56.850990 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 16:39:56.851145 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260619 16:39:56.852638 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4: Bootstrap starting.
I20260619 16:39:56.854528 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:56.855142 21920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:56.855892 21920 log.cc:824] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:56.857839 21920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4: No bootstrap required, opened a new log
I20260619 16:39:56.858330 21923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:56.856610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.15:55073 (local address 127.0.0.1:33305)
0619 16:39:56.856718 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:56.856724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:56.856908 (+ 184us) server_negotiation.cc:407] Connection header received
0619 16:39:56.857050 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:56.857055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:56.857096 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:56.857225 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:56.857954 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:56.857958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:56.857960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:56.857971 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:56.857996 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:56.858014 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:56.858102 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:56.858179 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 16:39:56.858218 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:39:56.861600 21920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30383773a2fb4b15b1f41d9e0832dee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33305 } }
I20260619 16:39:56.861939 21920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:56.862010 21920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30383773a2fb4b15b1f41d9e0832dee4, State: Initialized, Role: FOLLOWER
I20260619 16:39:56.862557 21920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30383773a2fb4b15b1f41d9e0832dee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33305 } }
I20260619 16:39:56.862756 21920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:56.862823 21920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:56.862924 21920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:56.864254 21920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30383773a2fb4b15b1f41d9e0832dee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33305 } }
I20260619 16:39:56.864634 21920 leader_election.cc:302] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30383773a2fb4b15b1f41d9e0832dee4; no voters:
I20260619 16:39:56.865021 21920 leader_election.cc:288] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:56.865129 21930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:56.865378 21930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [term 1 LEADER]: Becoming Leader. State: Replica: 30383773a2fb4b15b1f41d9e0832dee4, State: Running, Role: LEADER
I20260619 16:39:56.865666 21930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30383773a2fb4b15b1f41d9e0832dee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33305 } }
I20260619 16:39:56.866411 21920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:56.867233 21931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30383773a2fb4b15b1f41d9e0832dee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30383773a2fb4b15b1f41d9e0832dee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33305 } } }
I20260619 16:39:56.867357 21931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:56.867682 21939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:56.868870 21932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30383773a2fb4b15b1f41d9e0832dee4. Latest consensus state: current_term: 1 leader_uuid: "30383773a2fb4b15b1f41d9e0832dee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30383773a2fb4b15b1f41d9e0832dee4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33305 } } }
I20260619 16:39:56.868965 21932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:56.872080 21939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:56.877545 21939 catalog_manager.cc:1347] Generated new cluster ID: ff0cf5b7dfe049c98f9321662cb7a32b
I20260619 16:39:56.877612 21939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:56.886021 21939 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:56.886329 21939 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:56.894284 21939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30383773a2fb4b15b1f41d9e0832dee4: Generated new TSK 0
I20260619 16:39:56.894909 21939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:56.904232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21852
2026-06-19T16:39:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:56 dist-test-slave-0f28 krb5kdc[21951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:56 dist-test-slave-0f28 krb5kdc[21951](info): set up 2 sockets
Jun 19 16:39:56 dist-test-slave-0f28 krb5kdc[21951](info): commencing operation
krb5kdc: starting...
W20260619 16:39:57.961463 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:58 dist-test-slave-0f28 krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:39:58.080754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:58.190353 21964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:58.190608 21964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:58.190649 21964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:58.194875 21964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:58.194938 21964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:58.194958 21964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:58.194975 21964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:58.194990 21964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:58.200373 21964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:58.201447 21964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:58.202446 21964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:58.210654 21972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:58.210654 21969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:58.210672 21970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:58.210692 21964 server_base.cc:1034] running on GCE node
I20260619 16:39:58.211396 21964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:58.212414 21964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:58.213567 21964 hybrid_clock.cc:648] HybridClock initialized: now 1781887198213541 us; error 38 us; skew 500 ppm
Jun 19 16:39:58 dist-test-slave-0f28 krb5kdc[21951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:58.216223 21964 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:39:58.217185 21964 webserver.cc:460] Webserver started at http://127.0.98.62:43355/ using document root <none> and password file <none>
I20260619 16:39:58.217767 21964 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:58.217855 21964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:58.218067 21964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:58.220017 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "76a2e412de334b90baa90f1707a95f34"
format_stamp: "Formatted at 2026-06-19 16:39:58 on dist-test-slave-0f28"
server_key: "6b50eb6a44ad562063de0f72343730c6"
server_key_iv: "4b89ecfff80e1aa9af37c4884d3a8467"
server_key_version: "encryptionkey@0"
I20260619 16:39:58.220564 21964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "76a2e412de334b90baa90f1707a95f34"
format_stamp: "Formatted at 2026-06-19 16:39:58 on dist-test-slave-0f28"
server_key: "6b50eb6a44ad562063de0f72343730c6"
server_key_iv: "4b89ecfff80e1aa9af37c4884d3a8467"
server_key_version: "encryptionkey@0"
I20260619 16:39:58.224313 21964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 16:39:58.226796 21979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:58.228032 21964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 16:39:58.228150 21964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "76a2e412de334b90baa90f1707a95f34"
format_stamp: "Formatted at 2026-06-19 16:39:58 on dist-test-slave-0f28"
server_key: "6b50eb6a44ad562063de0f72343730c6"
server_key_iv: "4b89ecfff80e1aa9af37c4884d3a8467"
server_key_version: "encryptionkey@0"
I20260619 16:39:58.228277 21964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:58.238850 21964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:58.241972 21964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:58.242156 21964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:58.248853 21964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43647
I20260619 16:39:58.248920 22031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43647 every 8 connection(s)
I20260619 16:39:58.249830 21964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:58.252488 22032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:58.256078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 21964
I20260619 16:39:58.256198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:58.256491 392 external_mini_cluster.cc:1428] Setting key 417ac1406e877c0a49f425581e1d1aec
I20260619 16:39:58.257992 22032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34: Bootstrap starting.
I20260619 16:39:58.260313 22032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:58.261034 22032 log.cc:824] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:58.262920 22032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34: No bootstrap required, opened a new log
Jun 19 16:39:58 dist-test-slave-0f28 krb5kdc[21951](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:39:58.267153 22032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2e412de334b90baa90f1707a95f34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260619 16:39:58.267413 22032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:58.267478 22032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a2e412de334b90baa90f1707a95f34, State: Initialized, Role: FOLLOWER
I20260619 16:39:58.267961 22032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2e412de334b90baa90f1707a95f34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260619 16:39:58.268143 22032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:58.268210 22032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:58.268301 22032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:58.269129 22032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2e412de334b90baa90f1707a95f34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260619 16:39:58.269491 22032 leader_election.cc:302] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a2e412de334b90baa90f1707a95f34; no voters:
I20260619 16:39:58.269797 22032 leader_election.cc:288] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:58.270164 22038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:58.270370 22038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [term 1 LEADER]: Becoming Leader. State: Replica: 76a2e412de334b90baa90f1707a95f34, State: Running, Role: LEADER
I20260619 16:39:58.270663 22038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2e412de334b90baa90f1707a95f34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } }
I20260619 16:39:58.271049 22032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:58.271106 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:58.257903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58373 (local address 127.0.98.62:43647)
0619 16:39:58.258295 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:58.258310 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:58.258342 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:39:58.258937 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:58.258952 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:58.259272 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:58.259528 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:58.260474 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:58.261581 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:58.262421 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:58.263075 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:58.263247 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:58.265864 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:58.265870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:58.265887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:58.265916 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:58.267867 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:58.268465 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:58.268470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:58.268476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:58.268593 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:58.268930 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:58.268933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:58.268935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:58.269372 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:58.269600 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:58.269927 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 16:39:58.270172 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":131,"threads_started":1}
I20260619 16:39:58.271771 22038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a2e412de334b90baa90f1707a95f34. Latest consensus state: current_term: 1 leader_uuid: "76a2e412de334b90baa90f1707a95f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2e412de334b90baa90f1707a95f34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } } }
I20260619 16:39:58.271878 22038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:58.272238 22045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:58.272413 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a2e412de334b90baa90f1707a95f34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a2e412de334b90baa90f1707a95f34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43647 } } }
I20260619 16:39:58.272554 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:58.274077 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:58.275393 22045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:58.281294 22045 catalog_manager.cc:1347] Generated new cluster ID: 5462258325794b8aa193c4f074b1571b
I20260619 16:39:58.281380 22045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:58.287916 22035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:58.277971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34025 (local address 127.0.98.62:43647)
0619 16:39:58.278066 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:58.278070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:58.279021 (+ 951us) server_negotiation.cc:407] Connection header received
0619 16:39:58.279172 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:58.279175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:58.279216 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:58.279310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:58.279940 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:58.280680 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:58.282048 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:58.282388 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:58.282489 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:58.284166 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:58.284169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:58.284172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:39:58.284195 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:58.285886 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:58.286431 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:58.286434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:58.286436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:58.286515 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:39:58.286947 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:39:58.286951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:39:58.286953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:39:58.287157 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:58.287286 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:58.287667 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 16:39:58.287767 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:39:58.297628 22045 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:58.298861 22045 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:58.307565 22045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a2e412de334b90baa90f1707a95f34: Generated new TSK 0
I20260619 16:39:58.308240 22045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:58.317680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 21964
W20260619 16:39:58.323025 21949 connection.cc:535] client connection to 127.0.98.62:43647 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:43647): Connection reset by peer (error 104)
2026-06-19T16:39:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:58 dist-test-slave-0f28 krb5kdc[22063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:58 dist-test-slave-0f28 krb5kdc[22063](info): set up 2 sockets
Jun 19 16:39:58 dist-test-slave-0f28 krb5kdc[22063](info): commencing operation
krb5kdc: starting...
W20260619 16:39:59.389765 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:39:59 dist-test-slave-0f28 krb5kdc[22063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:39:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:39:59.508448 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44933
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:39:59.612692 22076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:39:59.612941 22076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:39:59.612979 22076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:39:59.616865 22076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:39:59.616919 22076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:39:59.616938 22076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:39:59.616954 22076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:39:59.616969 22076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:39:59.622076 22076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44933
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:39:59.622941 22076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:39:59.623811 22076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:39:59.632100 22081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:39:59.632149 22084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:59.632143 22076 server_base.cc:1034] running on GCE node
W20260619 16:39:59.632104 22082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:39:59.632776 22076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:39:59.633860 22076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:39:59.635017 22076 hybrid_clock.cc:648] HybridClock initialized: now 1781887199634989 us; error 42 us; skew 500 ppm
Jun 19 16:39:59 dist-test-slave-0f28 krb5kdc[22063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:39:59.637625 22076 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:39:59.638453 22076 webserver.cc:460] Webserver started at http://127.0.98.62:34063/ using document root <none> and password file <none>
I20260619 16:39:59.638971 22076 fs_manager.cc:359] Metadata directory not provided
I20260619 16:39:59.639055 22076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:39:59.639298 22076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:39:59.641153 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "455f5e6e327d423e99b42557d5d03d56"
format_stamp: "Formatted at 2026-06-19 16:39:59 on dist-test-slave-0f28"
server_key: "367fc6a7d6227aa2faa41825941caffb"
server_key_iv: "24876c4ef49fd136adeee4fd01914222"
server_key_version: "encryptionkey@0"
I20260619 16:39:59.641745 22076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "455f5e6e327d423e99b42557d5d03d56"
format_stamp: "Formatted at 2026-06-19 16:39:59 on dist-test-slave-0f28"
server_key: "367fc6a7d6227aa2faa41825941caffb"
server_key_iv: "24876c4ef49fd136adeee4fd01914222"
server_key_version: "encryptionkey@0"
I20260619 16:39:59.645066 22076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 16:39:59.647361 22091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:39:59.648494 22076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:39:59.648625 22076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "455f5e6e327d423e99b42557d5d03d56"
format_stamp: "Formatted at 2026-06-19 16:39:59 on dist-test-slave-0f28"
server_key: "367fc6a7d6227aa2faa41825941caffb"
server_key_iv: "24876c4ef49fd136adeee4fd01914222"
server_key_version: "encryptionkey@0"
I20260619 16:39:59.648727 22076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:39:59.670953 22076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:39:59.673810 22076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:39:59.674010 22076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:39:59.680968 22076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44933
I20260619 16:39:59.680990 22143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44933 every 8 connection(s)
I20260619 16:39:59.681912 22076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:39:59.683751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22076
I20260619 16:39:59.683887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:39:59.684198 392 external_mini_cluster.cc:1428] Setting key 1c55ec8dfc085088d08e320fbe3685d1
I20260619 16:39:59.684707 22144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:39:59.690260 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56: Bootstrap starting.
I20260619 16:39:59.692461 22144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56: Neither blocks nor log segments found. Creating new log.
I20260619 16:39:59.692677 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:59.685561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50387 (local address 127.0.98.62:44933)
0619 16:39:59.685975 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:59.685990 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:59.686022 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:39:59.686638 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:59.686653 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:59.686965 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:59.687278 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:59.687649 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:59.688586 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:59.689563 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:59.690029 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:59.690208 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:59.691031 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:59.691036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:59.691046 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:59.691417 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:59.691467 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:59.691488 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:59.691623 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:59.691815 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 16:39:59.691934 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":140,"threads_started":1}
I20260619 16:39:59.693221 22144 log.cc:824] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56: Log is configured to *not* fsync() on all Append() calls
I20260619 16:39:59.695288 22144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56: No bootstrap required, opened a new log
I20260619 16:39:59.695449 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:39:59.699331 22144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455f5e6e327d423e99b42557d5d03d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44933 } }
I20260619 16:39:59.699620 22144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:39:59.699690 22144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455f5e6e327d423e99b42557d5d03d56, State: Initialized, Role: FOLLOWER
I20260619 16:39:59.700114 22144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455f5e6e327d423e99b42557d5d03d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44933 } }
I20260619 16:39:59.700273 22144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:39:59.700351 22144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:39:59.700461 22144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:39:59.701257 22144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455f5e6e327d423e99b42557d5d03d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44933 } }
I20260619 16:39:59.701409 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:39:59.697267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56165 (local address 127.0.98.62:44933)
0619 16:39:59.697388 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:39:59.697392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:39:59.697560 (+ 168us) server_negotiation.cc:407] Connection header received
0619 16:39:59.697674 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:39:59.697676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:39:59.697709 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:39:59.697775 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:39:59.698033 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:59.698483 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:59.699601 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:39:59.699973 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:39:59.700064 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:39:59.700906 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:39:59.700909 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:39:59.700912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:39:59.700923 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:39:59.700955 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:39:59.700965 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:39:59.701059 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 16:39:59.701175 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:39:59.701208 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:39:59.701642 22144 leader_election.cc:302] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 455f5e6e327d423e99b42557d5d03d56; no voters:
I20260619 16:39:59.701946 22144 leader_election.cc:288] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:39:59.702060 22154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:39:59.702288 22154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [term 1 LEADER]: Becoming Leader. State: Replica: 455f5e6e327d423e99b42557d5d03d56, State: Running, Role: LEADER
I20260619 16:39:59.702634 22154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455f5e6e327d423e99b42557d5d03d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44933 } }
I20260619 16:39:59.703063 22144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:39:59.704066 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 455f5e6e327d423e99b42557d5d03d56. Latest consensus state: current_term: 1 leader_uuid: "455f5e6e327d423e99b42557d5d03d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455f5e6e327d423e99b42557d5d03d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44933 } } }
I20260619 16:39:59.704191 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:59.704553 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "455f5e6e327d423e99b42557d5d03d56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455f5e6e327d423e99b42557d5d03d56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44933 } } }
I20260619 16:39:59.704591 22163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:39:59.704697 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56 [sys.catalog]: This master's current role is: LEADER
I20260619 16:39:59.707381 22163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:39:59.712118 22163 catalog_manager.cc:1347] Generated new cluster ID: 07a0939aa54446c3aa85214079e1a592
I20260619 16:39:59.712186 22163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:39:59.732177 22163 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:39:59.733089 22163 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:39:59.738327 22163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 455f5e6e327d423e99b42557d5d03d56: Generated new TSK 0
I20260619 16:39:59.739006 22163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:39:59.745677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22076
W20260619 16:39:59.750870 22061 connection.cc:535] client connection to 127.0.98.62:44933 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44933): Connection reset by peer (error 104)
2026-06-19T16:39:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:39:59 dist-test-slave-0f28 krb5kdc[22175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:39:59 dist-test-slave-0f28 krb5kdc[22175](info): set up 2 sockets
Jun 19 16:39:59 dist-test-slave-0f28 krb5kdc[22175](info): commencing operation
krb5kdc: starting...
W20260619 16:40:00.793771 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:40:00 dist-test-slave-0f28 krb5kdc[22175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:40:00.916095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34851
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:40:01.024775 22188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:01.025009 22188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:01.025049 22188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:01.029021 22188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:01.029074 22188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:01.029093 22188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:01.029109 22188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:01.029174 22188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:01.034411 22188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:01.035418 22188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:01.036401 22188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:01.044759 22196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:01.044772 22194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:01.044773 22193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:01.045156 22188 server_base.cc:1034] running on GCE node
I20260619 16:40:01.045728 22188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:01.046798 22188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:01.047999 22188 hybrid_clock.cc:648] HybridClock initialized: now 1781887201047971 us; error 38 us; skew 500 ppm
Jun 19 16:40:01 dist-test-slave-0f28 krb5kdc[22175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:01.050792 22188 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:40:01.051728 22188 webserver.cc:460] Webserver started at http://127.0.98.62:33083/ using document root <none> and password file <none>
I20260619 16:40:01.052281 22188 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:01.052376 22188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:01.052600 22188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:01.054643 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "2d2900d1f2074fedadb673dd1c292bf5"
format_stamp: "Formatted at 2026-06-19 16:40:01 on dist-test-slave-0f28"
server_key: "138b14cf014272e556e3945fbcaf76a9"
server_key_iv: "d0d3bef892c8f024d70cd4ba30de7a7b"
server_key_version: "encryptionkey@0"
I20260619 16:40:01.055197 22188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d2900d1f2074fedadb673dd1c292bf5"
format_stamp: "Formatted at 2026-06-19 16:40:01 on dist-test-slave-0f28"
server_key: "138b14cf014272e556e3945fbcaf76a9"
server_key_iv: "d0d3bef892c8f024d70cd4ba30de7a7b"
server_key_version: "encryptionkey@0"
I20260619 16:40:01.058743 22188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:40:01.061044 22203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:01.062350 22188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:40:01.062491 22188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "2d2900d1f2074fedadb673dd1c292bf5"
format_stamp: "Formatted at 2026-06-19 16:40:01 on dist-test-slave-0f28"
server_key: "138b14cf014272e556e3945fbcaf76a9"
server_key_iv: "d0d3bef892c8f024d70cd4ba30de7a7b"
server_key_version: "encryptionkey@0"
I20260619 16:40:01.062600 22188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:01.067117 22188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:01.069931 22188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:01.070156 22188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:01.076817 22188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34851
I20260619 16:40:01.076844 22255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34851 every 8 connection(s)
I20260619 16:40:01.077721 22188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:40:01.080232 22256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:01.081938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22188
I20260619 16:40:01.082046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:40:01.082340 392 external_mini_cluster.cc:1428] Setting key 39a13ee52b6858cf7cc9be7596855c83
I20260619 16:40:01.086813 22256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5: Bootstrap starting.
I20260619 16:40:01.087567 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:01.083655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35025 (local address 127.0.98.62:34851)
0619 16:40:01.084044 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:01.084064 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:01.084105 (+ 41us) server_negotiation.cc:407] Connection header received
0619 16:40:01.084915 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:01.084937 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:01.085269 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:01.085655 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:01.086241 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:01.086245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:01.086258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:40:01.086268 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:01.086320 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:40:01.086344 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:01.086472 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:01.086630 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 16:40:01.086769 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":97,"threads_started":1}
I20260619 16:40:01.089214 22256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:01.090055 22256 log.cc:824] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:01.090374 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:40:01.091974 22256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5: No bootstrap required, opened a new log
I20260619 16:40:01.094024 22259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:01.092410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39239 (local address 127.0.98.62:34851)
0619 16:40:01.092511 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:01.092516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:01.092704 (+ 188us) server_negotiation.cc:407] Connection header received
0619 16:40:01.092820 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:01.092823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:01.092850 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:01.092941 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:01.093654 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:01.093657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:01.093660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:40:01.093671 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:01.093696 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:40:01.093707 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:01.093799 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:01.093881 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 16:40:01.093916 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 16:40:01.095969 22256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2900d1f2074fedadb673dd1c292bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260619 16:40:01.096174 22256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:40:01.096231 22256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2900d1f2074fedadb673dd1c292bf5, State: Initialized, Role: FOLLOWER
I20260619 16:40:01.096700 22256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2900d1f2074fedadb673dd1c292bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260619 16:40:01.096846 22256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:40:01.096912 22256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:40:01.097002 22256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:01.097895 22256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2900d1f2074fedadb673dd1c292bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260619 16:40:01.098217 22256 leader_election.cc:302] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d2900d1f2074fedadb673dd1c292bf5; no voters:
I20260619 16:40:01.098526 22256 leader_election.cc:288] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:40:01.098654 22266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:40:01.098877 22266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2900d1f2074fedadb673dd1c292bf5, State: Running, Role: LEADER
I20260619 16:40:01.099156 22266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2900d1f2074fedadb673dd1c292bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260619 16:40:01.099660 22256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:01.100910 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d2900d1f2074fedadb673dd1c292bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2900d1f2074fedadb673dd1c292bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } } }
I20260619 16:40:01.100903 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2900d1f2074fedadb673dd1c292bf5. Latest consensus state: current_term: 1 leader_uuid: "2d2900d1f2074fedadb673dd1c292bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2900d1f2074fedadb673dd1c292bf5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } } }
I20260619 16:40:01.101040 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:01.101089 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:01.101455 22275 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:40:01.103892 22275 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:40:01.109184 22275 catalog_manager.cc:1347] Generated new cluster ID: 79140ebd5b71489c97b5754b8bc88ce8
I20260619 16:40:01.109270 22275 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:40:01.121704 22275 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:40:01.122035 22275 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:40:01.126395 22275 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d2900d1f2074fedadb673dd1c292bf5: Generated new TSK 0
I20260619 16:40:01.127015 22275 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:40:01.134055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22188
2026-06-19T16:40:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:40:01 dist-test-slave-0f28 krb5kdc[22287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:40:01 dist-test-slave-0f28 krb5kdc[22287](info): set up 2 sockets
Jun 19 16:40:01 dist-test-slave-0f28 krb5kdc[22287](info): commencing operation
krb5kdc: starting...
W20260619 16:40:02.173622 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:40:02 dist-test-slave-0f28 krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:40:02.290557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42761
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:40:02.396457 22300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:02.396739 22300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:02.396818 22300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:02.401005 22300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:02.401109 22300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:02.401155 22300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:02.401249 22300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:02.401290 22300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:02.406580 22300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:02.407545 22300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:02.408488 22300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:02.416841 22305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:02.416814 22308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:02.416863 22306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:02.416962 22300 server_base.cc:1034] running on GCE node
I20260619 16:40:02.417625 22300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:02.418630 22300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:02.419788 22300 hybrid_clock.cc:648] HybridClock initialized: now 1781887202419773 us; error 33 us; skew 500 ppm
Jun 19 16:40:02 dist-test-slave-0f28 krb5kdc[22287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:02.427587 22300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:40:02.428438 22300 webserver.cc:460] Webserver started at http://127.0.98.62:37767/ using document root <none> and password file <none>
I20260619 16:40:02.428960 22300 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:02.429018 22300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:02.429215 22300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:02.431185 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "213c4d86a54d403ba3d99aaf5581fe5c"
format_stamp: "Formatted at 2026-06-19 16:40:02 on dist-test-slave-0f28"
server_key: "dcd7d22e16e428c71e2e198ea0edb9e7"
server_key_iv: "ffd4667cf6565c0919b9aefe7a2cd2da"
server_key_version: "encryptionkey@0"
I20260619 16:40:02.431736 22300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "213c4d86a54d403ba3d99aaf5581fe5c"
format_stamp: "Formatted at 2026-06-19 16:40:02 on dist-test-slave-0f28"
server_key: "dcd7d22e16e428c71e2e198ea0edb9e7"
server_key_iv: "ffd4667cf6565c0919b9aefe7a2cd2da"
server_key_version: "encryptionkey@0"
I20260619 16:40:02.435225 22300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:40:02.437485 22315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:02.438676 22300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:40:02.438812 22300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "213c4d86a54d403ba3d99aaf5581fe5c"
format_stamp: "Formatted at 2026-06-19 16:40:02 on dist-test-slave-0f28"
server_key: "dcd7d22e16e428c71e2e198ea0edb9e7"
server_key_iv: "ffd4667cf6565c0919b9aefe7a2cd2da"
server_key_version: "encryptionkey@0"
I20260619 16:40:02.438916 22300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:02.457990 22300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:02.460958 22300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:02.461154 22300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:02.468433 22300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42761
I20260619 16:40:02.468459 22367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42761 every 8 connection(s)
I20260619 16:40:02.469368 22300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:40:02.472021 22368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:02.476214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22300
I20260619 16:40:02.476363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:40:02.476656 392 external_mini_cluster.cc:1428] Setting key f6fdf8043cce02ed340433a48ac793cd
I20260619 16:40:02.477906 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c: Bootstrap starting.
I20260619 16:40:02.480386 22368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:02.481158 22368 log.cc:824] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:02.483430 22368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c: No bootstrap required, opened a new log
Jun 19 16:40:02 dist-test-slave-0f28 krb5kdc[22287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:02.487924 22368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42761 } }
I20260619 16:40:02.488198 22368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:40:02.488261 22368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 213c4d86a54d403ba3d99aaf5581fe5c, State: Initialized, Role: FOLLOWER
I20260619 16:40:02.488718 22368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42761 } }
I20260619 16:40:02.488893 22368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:40:02.488962 22368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:40:02.489058 22368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:02.490214 22368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42761 } }
I20260619 16:40:02.490588 22368 leader_election.cc:302] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 213c4d86a54d403ba3d99aaf5581fe5c; no voters:
I20260619 16:40:02.490933 22368 leader_election.cc:288] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:40:02.491034 22373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:40:02.491277 22373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [term 1 LEADER]: Becoming Leader. State: Replica: 213c4d86a54d403ba3d99aaf5581fe5c, State: Running, Role: LEADER
I20260619 16:40:02.491441 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:02.478065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37381 (local address 127.0.98.62:42761)
0619 16:40:02.478451 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:02.478467 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:02.478499 (+ 32us) server_negotiation.cc:407] Connection header received
0619 16:40:02.479109 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:02.479125 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:02.479446 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:02.479714 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:02.480710 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:02.481841 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:02.482802 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:02.483475 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:02.483691 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:02.486279 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:02.486284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:02.486296 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:02.486321 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:02.488381 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:02.488832 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:02.488839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:02.488847 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:02.488997 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:02.489285 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:02.489291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:02.489294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:02.489778 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:02.489971 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:02.490190 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 16:40:02.490485 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260619 16:40:02.491598 22373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42761 } }
I20260619 16:40:02.492165 22368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:02.493101 22375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 213c4d86a54d403ba3d99aaf5581fe5c. Latest consensus state: current_term: 1 leader_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42761 } } }
I20260619 16:40:02.493211 22375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:02.493469 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "213c4d86a54d403ba3d99aaf5581fe5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42761 } } }
I20260619 16:40:02.493546 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:02.494671 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:40:02.494575 22386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:40:02.497646 22386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:40:02.502672 22386 catalog_manager.cc:1347] Generated new cluster ID: 3884e6383b734c45bd8a51e46c48d57c
I20260619 16:40:02.502741 22386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:40:02.519644 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:02.496969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47817 (local address 127.0.98.62:42761)
0619 16:40:02.497061 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:02.497065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:02.511264 (+ 14199us) server_negotiation.cc:407] Connection header received
0619 16:40:02.511514 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:02.511517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:02.511559 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:02.511626 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:02.512543 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:02.513365 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:02.514427 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:02.514791 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:02.514893 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:02.516389 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:02.516393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:02.516396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:02.516421 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:02.518075 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:02.518508 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:02.518511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:02.518513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:02.518599 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:02.518849 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:02.518852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:02.518854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:02.519047 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:02.519177 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:02.519374 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 16:40:02.519489 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 16:40:02.533099 22386 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:40:02.534284 22386 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:40:02.541375 22386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 213c4d86a54d403ba3d99aaf5581fe5c: Generated new TSK 0
I20260619 16:40:02.542173 22386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:40:02.547276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22300
2026-06-19T16:40:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:40:02 dist-test-slave-0f28 krb5kdc[22399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:40:02 dist-test-slave-0f28 krb5kdc[22399](info): set up 2 sockets
Jun 19 16:40:02 dist-test-slave-0f28 krb5kdc[22399](info): commencing operation
krb5kdc: starting...
W20260619 16:40:03.601684 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:40:03 dist-test-slave-0f28 krb5kdc[22399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:40:03.720222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36393
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:40:03.826905 22412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:03.827186 22412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:03.827273 22412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:03.831429 22412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:03.831506 22412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:03.831548 22412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:03.831585 22412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:03.831621 22412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:03.836611 22412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:03.837605 22412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:03.838546 22412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:03.847195 22418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:03.847337 22412 server_base.cc:1034] running on GCE node
W20260619 16:40:03.847352 22420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:03.847242 22417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:03.847978 22412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:03.849042 22412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:03.850226 22412 hybrid_clock.cc:648] HybridClock initialized: now 1781887203850181 us; error 69 us; skew 500 ppm
Jun 19 16:40:03 dist-test-slave-0f28 krb5kdc[22399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:03.853008 22412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:40:03.853911 22412 webserver.cc:460] Webserver started at http://127.0.98.62:40237/ using document root <none> and password file <none>
I20260619 16:40:03.854457 22412 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:03.854548 22412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:03.854851 22412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:03.856824 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "bbf5c0e138e8433092028c301d79d980"
format_stamp: "Formatted at 2026-06-19 16:40:03 on dist-test-slave-0f28"
server_key: "85454ba92b6e0f47f65a7782dc0cba1f"
server_key_iv: "3ffe2458430dc118bbf715852bb39853"
server_key_version: "encryptionkey@0"
I20260619 16:40:03.857414 22412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "bbf5c0e138e8433092028c301d79d980"
format_stamp: "Formatted at 2026-06-19 16:40:03 on dist-test-slave-0f28"
server_key: "85454ba92b6e0f47f65a7782dc0cba1f"
server_key_iv: "3ffe2458430dc118bbf715852bb39853"
server_key_version: "encryptionkey@0"
I20260619 16:40:03.860965 22412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 16:40:03.863440 22427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:03.864638 22412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:40:03.864773 22412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "bbf5c0e138e8433092028c301d79d980"
format_stamp: "Formatted at 2026-06-19 16:40:03 on dist-test-slave-0f28"
server_key: "85454ba92b6e0f47f65a7782dc0cba1f"
server_key_iv: "3ffe2458430dc118bbf715852bb39853"
server_key_version: "encryptionkey@0"
I20260619 16:40:03.864880 22412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:03.883464 22412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:03.886322 22412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:03.886511 22412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:03.893291 22412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36393
I20260619 16:40:03.893340 22479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36393 every 8 connection(s)
I20260619 16:40:03.894186 22412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:40:03.895727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22412
I20260619 16:40:03.895879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:40:03.896202 392 external_mini_cluster.cc:1428] Setting key af6f61830144256ddc705da8f6269035
I20260619 16:40:03.896998 22480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:03.903709 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980: Bootstrap starting.
I20260619 16:40:03.905529 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:03.897805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34999 (local address 127.0.98.62:36393)
0619 16:40:03.898420 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:03.898436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:03.898471 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:40:03.899110 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:03.899126 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:03.899457 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:03.899738 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:03.900116 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:03.901093 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:03.902323 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:03.902783 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:03.902916 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:03.903765 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:03.903770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:03.903782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:40:03.904200 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:03.904252 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:40:03.904294 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:03.904368 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:03.904535 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 16:40:03.904750 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":313,"threads_started":1}
I20260619 16:40:03.906131 22480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:03.906900 22480 log.cc:824] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:03.908402 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:40:03.909082 22480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980: No bootstrap required, opened a new log
I20260619 16:40:03.913573 22480 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf5c0e138e8433092028c301d79d980" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36393 } }
I20260619 16:40:03.913808 22480 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:40:03.913887 22480 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf5c0e138e8433092028c301d79d980, State: Initialized, Role: FOLLOWER
I20260619 16:40:03.914376 22480 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf5c0e138e8433092028c301d79d980" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36393 } }
I20260619 16:40:03.914525 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:03.910540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50297 (local address 127.0.98.62:36393)
0619 16:40:03.910642 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:03.910647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:03.910773 (+ 126us) server_negotiation.cc:407] Connection header received
0619 16:40:03.910840 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:03.910843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:03.910883 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:03.910969 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:03.911206 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:03.911987 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:03.912828 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:03.913169 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:03.913284 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:03.914140 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:03.914144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:03.914146 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:40:03.914158 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:03.914184 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:40:03.914197 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:03.914265 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:03.914348 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 16:40:03.914402 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:40:03.914563 22480 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:40:03.914625 22480 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:40:03.914716 22480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:03.915693 22480 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf5c0e138e8433092028c301d79d980" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36393 } }
I20260619 16:40:03.915966 22480 leader_election.cc:302] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf5c0e138e8433092028c301d79d980; no voters:
I20260619 16:40:03.916203 22480 leader_election.cc:288] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:40:03.916365 22490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:40:03.916754 22490 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [term 1 LEADER]: Becoming Leader. State: Replica: bbf5c0e138e8433092028c301d79d980, State: Running, Role: LEADER
I20260619 16:40:03.917121 22490 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf5c0e138e8433092028c301d79d980" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36393 } }
I20260619 16:40:03.917559 22480 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:03.918823 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf5c0e138e8433092028c301d79d980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf5c0e138e8433092028c301d79d980" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36393 } } }
I20260619 16:40:03.918962 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:03.919322 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf5c0e138e8433092028c301d79d980. Latest consensus state: current_term: 1 leader_uuid: "bbf5c0e138e8433092028c301d79d980" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf5c0e138e8433092028c301d79d980" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36393 } } }
I20260619 16:40:03.919417 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980 [sys.catalog]: This master's current role is: LEADER
W20260619 16:40:03.922461 22505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:40:03.922531 22505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:40:03.922616 22501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:40:03.923305 22501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:40:03.928329 22501 catalog_manager.cc:1347] Generated new cluster ID: 09ed5b1f2d704f16b8d56d27aac4777d
I20260619 16:40:03.928392 22501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:40:03.939491 22501 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:40:03.940682 22501 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:40:03.948217 22501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf5c0e138e8433092028c301d79d980: Generated new TSK 0
I20260619 16:40:03.948951 22501 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:40:03.953832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22412
2026-06-19T16:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:40:04 dist-test-slave-0f28 krb5kdc[22511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:40:04 dist-test-slave-0f28 krb5kdc[22511](info): set up 2 sockets
Jun 19 16:40:04 dist-test-slave-0f28 krb5kdc[22511](info): commencing operation
krb5kdc: starting...
W20260619 16:40:05.005637 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:40:05 dist-test-slave-0f28 krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:40:05.128894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42991
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:40:05.239256 22524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:05.239522 22524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:05.239566 22524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:05.243595 22524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:05.243695 22524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:05.243733 22524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:05.243795 22524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:05.243834 22524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:05.249070 22524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:05.250134 22524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:05.251174 22524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:05.259485 22529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:05.259487 22532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:05.259487 22530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:05.259697 22524 server_base.cc:1034] running on GCE node
I20260619 16:40:05.260545 22524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:05.261617 22524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:05.262804 22524 hybrid_clock.cc:648] HybridClock initialized: now 1781887205262763 us; error 45 us; skew 500 ppm
Jun 19 16:40:05 dist-test-slave-0f28 krb5kdc[22511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:05.265583 22524 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:40:05.266528 22524 webserver.cc:460] Webserver started at http://127.0.98.62:39533/ using document root <none> and password file <none>
I20260619 16:40:05.267117 22524 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:05.267179 22524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:05.267390 22524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:05.269527 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "ced93e4c66e84db7bbbeab905d15f3d1"
format_stamp: "Formatted at 2026-06-19 16:40:05 on dist-test-slave-0f28"
server_key: "6fa0e6feea15eb57d616ce132589b439"
server_key_iv: "b2e97c9fdd626fbde322be47260e4e71"
server_key_version: "encryptionkey@0"
I20260619 16:40:05.270108 22524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "ced93e4c66e84db7bbbeab905d15f3d1"
format_stamp: "Formatted at 2026-06-19 16:40:05 on dist-test-slave-0f28"
server_key: "6fa0e6feea15eb57d616ce132589b439"
server_key_iv: "b2e97c9fdd626fbde322be47260e4e71"
server_key_version: "encryptionkey@0"
I20260619 16:40:05.273883 22524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:40:05.276319 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:05.277608 22524 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:40:05.277748 22524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "ced93e4c66e84db7bbbeab905d15f3d1"
format_stamp: "Formatted at 2026-06-19 16:40:05 on dist-test-slave-0f28"
server_key: "6fa0e6feea15eb57d616ce132589b439"
server_key_iv: "b2e97c9fdd626fbde322be47260e4e71"
server_key_version: "encryptionkey@0"
I20260619 16:40:05.277855 22524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:05.282194 22524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:05.285166 22524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:05.285391 22524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:05.292183 22524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42991
I20260619 16:40:05.292212 22591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42991 every 8 connection(s)
I20260619 16:40:05.293042 22524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:40:05.294605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22524
I20260619 16:40:05.294737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:40:05.295034 392 external_mini_cluster.cc:1428] Setting key 458accd4c03fc17dfc3ce4390fa39e13
I20260619 16:40:05.296204 22592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:05.300122 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:05.296430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34991 (local address 127.0.98.62:42991)
0619 16:40:05.296861 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:05.296879 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:05.296914 (+ 35us) server_negotiation.cc:407] Connection header received
0619 16:40:05.297613 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:05.297629 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:05.297900 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:05.298225 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:05.298989 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:05.298994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:05.299004 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:40:05.299014 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:05.299061 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:40:05.299080 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:05.299221 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:05.299291 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 16:40:05.299440 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260619 16:40:05.302217 22592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1: Bootstrap starting.
I20260619 16:40:05.302906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:40:05.304863 22592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:05.305897 22592 log.cc:824] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:05.306748 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:05.305025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39251 (local address 127.0.98.62:42991)
0619 16:40:05.305134 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:05.305138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:05.305366 (+ 228us) server_negotiation.cc:407] Connection header received
0619 16:40:05.305493 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:05.305496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:05.305527 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:05.305618 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:05.306289 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:05.306292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:05.306295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 16:40:05.306307 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:05.306334 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 16:40:05.306347 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:05.306427 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:05.306572 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 16:40:05.306602 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:40:05.307842 22592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1: No bootstrap required, opened a new log
I20260619 16:40:05.311220 22592 raft_consensus.cc:348] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260619 16:40:05.311445 22592 raft_consensus.cc:374] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 16:40:05.311535 22592 raft_consensus.cc:729] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ced93e4c66e84db7bbbeab905d15f3d1, State: Initialized, Role: FOLLOWER
I20260619 16:40:05.311919 22592 consensus_queue.cc:260] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260619 16:40:05.312064 22592 raft_consensus.cc:388] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 16:40:05.312151 22592 raft_consensus.cc:482] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 16:40:05.312245 22592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:05.313066 22592 raft_consensus.cc:504] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260619 16:40:05.313448 22592 leader_election.cc:302] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ced93e4c66e84db7bbbeab905d15f3d1; no voters:
I20260619 16:40:05.313773 22592 leader_election.cc:288] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 16:40:05.313853 22602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:40:05.314059 22602 raft_consensus.cc:686] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [term 1 LEADER]: Becoming Leader. State: Replica: ced93e4c66e84db7bbbeab905d15f3d1, State: Running, Role: LEADER
I20260619 16:40:05.314431 22602 consensus_queue.cc:237] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } }
I20260619 16:40:05.315021 22592 sys_catalog.cc:564] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:05.316239 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ced93e4c66e84db7bbbeab905d15f3d1. Latest consensus state: current_term: 1 leader_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } } }
I20260619 16:40:05.316248 22603 sys_catalog.cc:455] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ced93e4c66e84db7bbbeab905d15f3d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42991 } } }
I20260619 16:40:05.316367 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:05.316365 22603 sys_catalog.cc:458] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:05.316792 22611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:40:05.319303 22611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:40:05.324535 22611 catalog_manager.cc:1347] Generated new cluster ID: 658df3ff2e81441eb58f6d1e182cd7c2
I20260619 16:40:05.324594 22611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:40:05.333829 22611 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:40:05.334122 22611 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:40:05.343962 22611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ced93e4c66e84db7bbbeab905d15f3d1: Generated new TSK 0
I20260619 16:40:05.344712 22611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 16:40:05.358954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22524
2026-06-19T16:40:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1404 ms)
[----------] 18 tests from AuthTokenIssuingTest (25300 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:40:05 dist-test-slave-0f28 krb5kdc[22623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:40:05 dist-test-slave-0f28 krb5kdc[22623](info): set up 2 sockets
Jun 19 16:40:05 dist-test-slave-0f28 krb5kdc[22623](info): commencing operation
krb5kdc: starting...
W20260619 16:40:06.401780 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:40:06.521728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41317
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41317,127.0.98.61:35863,127.0.98.60:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:40:06.626820 22636 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:40:06.627133 22636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:06.627194 22636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:06.627274 22636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:06.631227 22636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:06.631302 22636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:06.631346 22636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:06.631402 22636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:06.631439 22636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:06.636447 22636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41317,127.0.98.61:35863,127.0.98.60:45091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:06.637538 22636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:06.638499 22636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:06.646633 22641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:06.646689 22644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:06.646788 22636 server_base.cc:1034] running on GCE node
W20260619 16:40:06.646692 22642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:06.647534 22636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:06.648561 22636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:06.649715 22636 hybrid_clock.cc:648] HybridClock initialized: now 1781887206649703 us; error 27 us; skew 500 ppm
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:06.652392 22636 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:40:06.653262 22636 webserver.cc:460] Webserver started at http://127.0.98.62:34365/ using document root <none> and password file <none>
I20260619 16:40:06.653825 22636 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:06.653894 22636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:06.654052 22636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:06.655923 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "ef761a3cf98c42339eb6ed53c7f61ed1"
format_stamp: "Formatted at 2026-06-19 16:40:06 on dist-test-slave-0f28"
server_key: "03e887033f780cf71d84455789b91410"
server_key_iv: "cd920948a0ff8631fdf947c40ee67485"
server_key_version: "encryptionkey@0"
I20260619 16:40:06.656428 22636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef761a3cf98c42339eb6ed53c7f61ed1"
format_stamp: "Formatted at 2026-06-19 16:40:06 on dist-test-slave-0f28"
server_key: "03e887033f780cf71d84455789b91410"
server_key_iv: "cd920948a0ff8631fdf947c40ee67485"
server_key_version: "encryptionkey@0"
I20260619 16:40:06.659948 22636 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 16:40:06.662268 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:06.663463 22636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:40:06.663576 22636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "ef761a3cf98c42339eb6ed53c7f61ed1"
format_stamp: "Formatted at 2026-06-19 16:40:06 on dist-test-slave-0f28"
server_key: "03e887033f780cf71d84455789b91410"
server_key_iv: "cd920948a0ff8631fdf947c40ee67485"
server_key_version: "encryptionkey@0"
I20260619 16:40:06.663731 22636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:06.689568 22636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:06.692984 22636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:06.693186 22636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:06.699864 22636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41317
I20260619 16:40:06.700139 22703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41317 every 8 connection(s)
I20260619 16:40:06.700796 22636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:40:06.703514 22704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:06.707242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22636
I20260619 16:40:06.707379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:40:06.707641 392 external_mini_cluster.cc:1428] Setting key 29c2ad29155226dd37ae6f7da3933e3a
I20260619 16:40:06.707360 22704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:06.725553 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.709584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41317 (local address 127.0.0.1:35174)
0619 16:40:06.710023 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.710075 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:06.711254 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:06.711953 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:06.711961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:06.712461 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:06.712858 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.712894 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.714129 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.714132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.715354 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.715367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.716033 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.716037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.716159 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.716789 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:06.716810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:06.720198 (+ 3388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:06.722475 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.722483 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.722491 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.722724 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.723157 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.723161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.723164 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.723298 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.724121 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:06.724126 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:06.724248 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 16:40:06.724398 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 16:40:06.724672 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260619 16:40:06.725556 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.709584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.0.98.62:41317)
0619 16:40:06.710162 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:06.710173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:06.710221 (+ 48us) server_negotiation.cc:407] Connection header received
0619 16:40:06.711557 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:06.711562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:06.711731 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:06.711933 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:06.713062 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.713995 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.715501 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.715864 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.715984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.720367 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:06.720372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:06.720382 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:06.720403 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:06.722360 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.722844 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.722853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.722858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.723003 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.723406 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.723410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.723412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.723950 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:06.724105 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:06.724517 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 16:40:06.724810 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":453,"spinlock_wait_cycles":10112,"thread_start_us":167,"threads_started":1}
I20260619 16:40:06.728066 22704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:06.728706 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.728484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46258)
0619 16:40:06.728590 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.728655 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:40:06.728940 22654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
I20260619 16:40:06.729915 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.729655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46264)
0619 16:40:06.729783 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.729843 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 16:40:06.730058 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35863: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:40:06.736346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35863
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41317,127.0.98.61:35863,127.0.98.60:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:40:06.779333 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } attempt: 1
I20260619 16:40:06.780210 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.779882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46268)
0619 16:40:06.780052 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.780123 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 16:40:06.780928 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.780708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46280)
0619 16:40:06.780819 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.780869 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 16:40:06.781064 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35863: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
W20260619 16:40:06.844832 22711 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:40:06.845121 22711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:06.845146 22711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:06.845183 22711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:06.849133 22711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:06.849220 22711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:06.849254 22711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:06.849277 22711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:06.849298 22711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:06.849225 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } attempt: 2
I20260619 16:40:06.850050 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.849730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46286)
0619 16:40:06.849892 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.849964 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:40:06.850788 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.850553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46296)
0619 16:40:06.850675 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.850726 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:40:06.850929 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35863: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35863: connect: Connection refused (error 111)
I20260619 16:40:06.854913 22711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41317,127.0.98.61:35863,127.0.98.60:45091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:06.855847 22711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:06.856773 22711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:06.866645 22717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:06.866792 22716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:06.866660 22719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:06.867267 22711 server_base.cc:1034] running on GCE node
I20260619 16:40:06.868002 22711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:06.869526 22711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:06.870708 22711 hybrid_clock.cc:648] HybridClock initialized: now 1781887206870692 us; error 34 us; skew 500 ppm
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:06.873657 22711 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:40:06.874567 22711 webserver.cc:460] Webserver started at http://127.0.98.61:43087/ using document root <none> and password file <none>
I20260619 16:40:06.875156 22711 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:06.875249 22711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:06.875452 22711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:06.877516 22711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data/instance:
uuid: "aebb88973e8c437c932d88ccc0efc562"
format_stamp: "Formatted at 2026-06-19 16:40:06 on dist-test-slave-0f28"
server_key: "1e31f347c141d424a961da038a9c879d"
server_key_iv: "e3afb63ff353594d981336d3f0202d59"
server_key_version: "encryptionkey@0"
I20260619 16:40:06.878077 22711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal/instance:
uuid: "aebb88973e8c437c932d88ccc0efc562"
format_stamp: "Formatted at 2026-06-19 16:40:06 on dist-test-slave-0f28"
server_key: "1e31f347c141d424a961da038a9c879d"
server_key_iv: "e3afb63ff353594d981336d3f0202d59"
server_key_version: "encryptionkey@0"
I20260619 16:40:06.881687 22711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 16:40:06.884043 22726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:06.885206 22711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:40:06.885411 22711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal
uuid: "aebb88973e8c437c932d88ccc0efc562"
format_stamp: "Formatted at 2026-06-19 16:40:06 on dist-test-slave-0f28"
server_key: "1e31f347c141d424a961da038a9c879d"
server_key_iv: "e3afb63ff353594d981336d3f0202d59"
server_key_version: "encryptionkey@0"
I20260619 16:40:06.885524 22711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:06.896317 22711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:06.899731 22711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:06.899947 22711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:06.907028 22711 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35863
I20260619 16:40:06.907336 22778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35863 every 8 connection(s)
I20260619 16:40:06.907998 22711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:40:06.910748 22779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:06.912073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22711
I20260619 16:40:06.912204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-1/wal/instance
I20260619 16:40:06.912501 392 external_mini_cluster.cc:1428] Setting key 341bd96deb6bfe0e834bf029a0b6adb7
I20260619 16:40:06.915293 22779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:06.930013 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.917186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.98.62:41317)
0619 16:40:06.917393 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:06.917400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:06.918200 (+ 800us) server_negotiation.cc:407] Connection header received
0619 16:40:06.919449 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:06.919452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:06.919502 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:06.919578 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:06.921083 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.921785 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.922991 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.923306 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.923401 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.926036 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:06.926039 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:06.926041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:06.926064 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:06.927815 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.928384 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.928388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.928389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.928472 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.928828 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.928832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.928834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.929019 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:06.929137 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:06.929711 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 16:40:06.929838 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:40:06.930776 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.917538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41317 (local address 127.0.0.1:35180)
0619 16:40:06.918022 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.918076 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:06.919214 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:06.919789 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:06.919799 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:06.920265 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:06.920941 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.920957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.921920 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.921923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.922841 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.922850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.923472 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.923481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.923626 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.924244 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:06.924261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:06.925863 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:06.927966 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.927971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.927986 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.928270 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.928573 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.928577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.928579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.928726 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.929137 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:06.929145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:06.929455 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 16:40:06.929652 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 16:40:06.929962 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":93,"threads_started":1}
I20260619 16:40:06.932384 22779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:40:06.940130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45091
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41317,127.0.98.61:35863,127.0.98.60:45091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:40:06.941836 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.932942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46298)
0619 16:40:06.933062 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.933078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:06.933160 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:06.933782 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:06.933785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:06.933916 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:06.933999 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.934004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.935096 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.935099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.935666 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.935674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.936359 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.936362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.936445 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.936906 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:06.936918 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:06.938347 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:06.940251 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.940257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.940262 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.940516 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.940802 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.940807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.940812 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.940914 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.941367 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:06.941372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:06.941493 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 16:40:06.941623 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 16:40:06.941723 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:40:06.942108 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.933125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.98.61:35863)
0619 16:40:06.933448 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:06.933457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:06.933476 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:40:06.933509 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:06.933513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:06.933665 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:06.933771 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:06.934099 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.934979 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.935777 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.936226 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.936314 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.938489 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:06.938514 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:06.938524 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:06.938545 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:06.940134 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.940633 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.940638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.940639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.940713 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.941003 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.941005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.941006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.941227 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:06.941390 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:06.941726 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 16:40:06.941885 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260619 16:40:06.942063 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } attempt: 3
I20260619 16:40:06.944056 22779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:06.944758 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.944501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34264)
0619 16:40:06.944654 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.944711 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260619 16:40:06.945367 22729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
I20260619 16:40:06.946151 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.945953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34268)
0619 16:40:06.946052 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.946106 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:40:06.946362 22779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45091: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Jun 19 16:40:06 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:40:06.952234 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.942463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46306)
0619 16:40:06.942633 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.942647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:06.942719 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:06.943066 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:06.943071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:06.943269 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:06.943406 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.943414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.944309 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.944316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.946115 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:06.946129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.946842 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.946848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:06.946990 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.947407 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:06.947427 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:06.948771 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:06.950838 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.950841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.950842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.951023 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.951333 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:06.951335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:06.951337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:06.951412 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:06.951858 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:06.951860 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:06.951933 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 16:40:06.952021 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:40:06.952122 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 16:40:06.952507 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.942548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.0.98.61:35863)
0619 16:40:06.942843 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:06.942849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:06.942860 (+ 11us) server_negotiation.cc:407] Connection header received
0619 16:40:06.942907 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:06.942910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:06.942951 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:06.943038 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:06.943528 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.944176 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.946399 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:06.946718 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:06.946800 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:06.948919 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:06.948927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:06.948932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:06.948963 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:06.950726 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.951123 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.951130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.951135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.951231 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:06.951514 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:06.951519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:06.951521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:06.951758 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:06.951871 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:06.952112 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 16:40:06.952244 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":157,"threads_started":1}
I20260619 16:40:06.953125 22704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:06.953770 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.953537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34274)
0619 16:40:06.953651 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.953704 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 16:40:06.954399 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.954159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34278)
0619 16:40:06.954295 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.954342 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 16:40:06.954520 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45091: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
I20260619 16:40:06.985689 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } attempt: 1
I20260619 16:40:06.986570 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.986222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34288)
0619 16:40:06.986399 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.986470 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:40:06.987282 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.987073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34298)
0619 16:40:06.987175 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.987224 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:40:06.987407 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45091: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
I20260619 16:40:06.995672 22779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } attempt: 1
I20260619 16:40:06.996413 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.996147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34306)
0619 16:40:06.996278 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.996344 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:40:06.997184 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:06.996948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34316)
0619 16:40:06.997073 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:06.997123 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:40:06.997367 22779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45091: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
W20260619 16:40:07.053342 22785 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:40:07.053685 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:07.053723 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:07.053761 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:07.057612 22785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:07.057659 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:07.057678 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:07.057694 22785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:07.057709 22785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:07.062578 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } attempt: 2
I20260619 16:40:07.062772 22785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41317,127.0.98.61:35863,127.0.98.60:45091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:07.063443 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.063117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34328)
0619 16:40:07.063282 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.063353 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 16:40:07.063980 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:07.064231 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.063988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34344)
0619 16:40:07.064116 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.064170 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 16:40:07.064357 22704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45091: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
I20260619 16:40:07.065065 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:40:07.065567 22779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } attempt: 2
I20260619 16:40:07.066231 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.065982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34350)
0619 16:40:07.066102 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.066157 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 16:40:07.067127 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.066832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34354)
0619 16:40:07.066971 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.067061 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 16:40:07.067272 22779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45091: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45091: connect: Connection refused (error 111)
W20260619 16:40:07.073549 22795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:07.073575 22793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:07.073614 22785 server_base.cc:1034] running on GCE node
W20260619 16:40:07.073561 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:07.074360 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:07.075332 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:07.076505 22785 hybrid_clock.cc:648] HybridClock initialized: now 1781887207076474 us; error 42 us; skew 500 ppm
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:07.079277 22785 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:40:07.080113 22785 webserver.cc:460] Webserver started at http://127.0.98.60:43563/ using document root <none> and password file <none>
I20260619 16:40:07.080663 22785 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:07.080754 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:07.080976 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:07.083468 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data/instance:
uuid: "00b7eeac482b42feb0d179e1d7fcdc9b"
format_stamp: "Formatted at 2026-06-19 16:40:07 on dist-test-slave-0f28"
server_key: "516bbebc1cf4395b9cd0b476ee5ef0d2"
server_key_iv: "16e52cdc0f9c97504cf4aacf69b841ab"
server_key_version: "encryptionkey@0"
I20260619 16:40:07.084033 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal/instance:
uuid: "00b7eeac482b42feb0d179e1d7fcdc9b"
format_stamp: "Formatted at 2026-06-19 16:40:07 on dist-test-slave-0f28"
server_key: "516bbebc1cf4395b9cd0b476ee5ef0d2"
server_key_iv: "16e52cdc0f9c97504cf4aacf69b841ab"
server_key_version: "encryptionkey@0"
I20260619 16:40:07.087781 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:40:07.090422 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:07.091651 22785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:40:07.091785 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal
uuid: "00b7eeac482b42feb0d179e1d7fcdc9b"
format_stamp: "Formatted at 2026-06-19 16:40:07 on dist-test-slave-0f28"
server_key: "516bbebc1cf4395b9cd0b476ee5ef0d2"
server_key_iv: "16e52cdc0f9c97504cf4aacf69b841ab"
server_key_version: "encryptionkey@0"
I20260619 16:40:07.091892 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:07.117098 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:07.120200 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:07.120416 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:07.127395 22785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45091
I20260619 16:40:07.127760 22854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45091 every 8 connection(s)
I20260619 16:40:07.128340 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:40:07.130995 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:07.134634 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:07.137984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 22785
I20260619 16:40:07.138099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781886927079793-392-0/minicluster-data/master-2/wal/instance
I20260619 16:40:07.138365 392 external_mini_cluster.cc:1428] Setting key 7b41949636de1371b6fa9e5cc474daf8
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:40:07.149121 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.136491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.0.98.62:41317)
0619 16:40:07.136626 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.136629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.137467 (+ 838us) server_negotiation.cc:407] Connection header received
0619 16:40:07.138611 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.138614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.138669 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.138781 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.140259 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.140843 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.142147 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.142353 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.142431 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.145245 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.145252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.145257 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.145288 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.147029 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.147574 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.147578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.147579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.147662 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.148089 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.148093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.148094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.148286 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.148397 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.148818 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.148935 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:40:07.149966 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.139060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.0.98.60:45091)
0619 16:40:07.139435 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.139443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.139462 (+ 19us) server_negotiation.cc:407] Connection header received
0619 16:40:07.139621 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.139627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.139796 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.139930 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.140756 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.141719 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.142714 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.143143 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.143257 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.145808 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.145813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.145822 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.145842 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.147416 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.147875 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.147881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.147883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.147971 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.148260 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.148264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.148266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.148595 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.148797 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.149007 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.149150 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":136,"threads_started":1}
I20260619 16:40:07.149966 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.136803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41317 (local address 127.0.0.1:35192)
0619 16:40:07.137257 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.137297 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:07.138367 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:07.138968 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:07.138979 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:07.139487 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:07.140070 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.140091 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.141007 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.141010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.142009 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.142019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.142497 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.142501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.142713 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.143369 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:07.143388 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:07.145086 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:07.147158 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.147168 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.147177 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.147456 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.147764 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.147768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.147770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.147963 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.148395 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:07.148402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:07.148603 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 16:40:07.148762 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 16:40:07.149065 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260619 16:40:07.151708 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:07.158442 22779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } attempt: 3
I20260619 16:40:07.162004 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.152152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35863 (local address 127.0.0.1:46314)
0619 16:40:07.152425 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.152440 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:07.152498 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:07.152744 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:07.152747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:07.152913 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:07.153062 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.153068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.154322 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.154326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.155172 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.155181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.155769 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.155773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.155861 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.156390 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:07.156410 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:07.157807 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:07.160337 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.160340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.160342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.160560 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.160896 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.160899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.160900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.160995 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.161517 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:07.161520 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:07.161617 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 16:40:07.161729 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:40:07.161846 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260619 16:40:07.162160 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.152404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.0.98.61:35863)
0619 16:40:07.152526 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.152530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.152551 (+ 21us) server_negotiation.cc:407] Connection header received
0619 16:40:07.152595 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.152597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.152635 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.152699 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.153213 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.154191 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.155286 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.155671 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.155761 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.157953 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.157957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.157959 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.157982 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.160209 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.160708 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.160712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.160714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.160800 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.161099 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.161103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.161105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.161415 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.161514 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.161809 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.161940 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":45}
I20260619 16:40:07.162722 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.152106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.98.62:41317)
0619 16:40:07.152222 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.152227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.152239 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:40:07.152361 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.152364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.152405 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.152476 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.153445 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.154201 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.155286 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.155622 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.155746 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.158577 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.158580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.158582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.158635 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.160535 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.160988 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.160991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.160993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.161068 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.161378 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.161382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.161386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.161621 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.161837 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.162435 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.162570 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:40:07.163084 22855 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:07.163595 22704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } attempt: 3
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:40:07.168537 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.158877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34376)
0619 16:40:07.159036 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.159052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:07.159171 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:07.159750 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:07.159753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:07.160110 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:07.160220 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.160225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.161219 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.161222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.162134 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.162146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.162942 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.162946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.163033 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.163479 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:07.163493 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:07.164976 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:07.166960 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.166963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.166965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.167213 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.167506 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.167512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.167513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.167597 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.168015 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:07.168018 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:07.168102 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 16:40:07.168200 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 16:40:07.168333 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":17}
I20260619 16:40:07.169198 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.159386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34376 (local address 127.0.98.60:45091)
0619 16:40:07.159527 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.159533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.159549 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:40:07.159587 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.159592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.159647 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.159734 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.160336 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.161086 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.162494 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.162812 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.163245 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.165120 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.165123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.165125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.165145 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.166840 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.167319 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.167322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.167324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.167402 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.167688 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.167691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.167693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.167878 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.167960 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.168878 (+ 918us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.169011 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:40:07 dist-test-slave-0f28 krb5kdc[22623](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:40:07.172896 22779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: Bootstrap starting.
I20260619 16:40:07.174803 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.163907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34400)
0619 16:40:07.164073 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.164085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:07.164171 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:07.165277 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:07.165282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:07.165502 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:07.165588 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.165595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.166924 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.166928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.167776 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.167786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.168497 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.168501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.168613 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.169101 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:07.169115 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:07.170895 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:07.173037 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.173040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.173042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.173305 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.173640 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.173643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.173646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.173743 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.174316 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:07.174319 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:07.174429 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 16:40:07.174517 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 16:40:07.174655 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 16:40:07.175391 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.163765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45091 (local address 127.0.0.1:34384)
0619 16:40:07.164766 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:07.164779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:07.164857 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:07.165121 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:07.165124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:07.165503 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:07.165588 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.165595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.167017 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.167020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.167793 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:07.167801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.168490 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.168493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:07.168564 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.168931 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:07.168940 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:07.171427 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:07.173998 (+ 2571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.174004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.174007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.174195 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.174482 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:07.174487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:07.174489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:07.174568 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:07.174914 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:07.174916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:07.174994 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 16:40:07.175096 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 16:40:07.175233 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933}
I20260619 16:40:07.175436 22860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.163967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34384 (local address 127.0.98.60:45091)
0619 16:40:07.164446 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.164454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.164848 (+ 394us) server_negotiation.cc:407] Connection header received
0619 16:40:07.164938 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.164941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.164999 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.165290 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.165713 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.166926 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.167935 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.168375 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.168759 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.172079 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.172082 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.172085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.172119 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.173905 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.174305 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.174308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.174310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.174394 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.174658 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.174662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.174664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.174818 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.174913 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.175148 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.175284 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":396,"thread_start_us":117,"threads_started":1}
I20260619 16:40:07.175880 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.164189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.0.98.60:45091)
0619 16:40:07.165062 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.165071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.165081 (+ 10us) server_negotiation.cc:407] Connection header received
0619 16:40:07.165105 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.165108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.165144 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.165705 (+ 561us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.165910 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.166807 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.167946 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.168375 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.168772 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.171041 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.171045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.171048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.171072 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.172911 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.173428 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.173431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.173433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.173528 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.173856 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.173859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.173861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.174191 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.175412 (+ 1221us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.175615 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.175725 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":802,"thread_start_us":89,"threads_started":1}
I20260619 16:40:07.176478 22779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:07.177635 22779 log.cc:824] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:07.178783 22855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: Bootstrap starting.
I20260619 16:40:07.179989 22779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: No bootstrap required, opened a new log
I20260619 16:40:07.180665 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.163956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.0.98.61:35863)
0619 16:40:07.164054 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.164060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.164216 (+ 156us) server_negotiation.cc:407] Connection header received
0619 16:40:07.164401 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.164404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.164443 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.164534 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.165264 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.166370 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.167523 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.167839 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.167924 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.177488 (+ 9564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.177491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.177494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.177516 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.179029 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.179469 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.179473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.179475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.179555 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.179835 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.179840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.179842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.180022 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.180216 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.180396 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.180513 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:40:07.181618 22855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:07.182514 22855 log.cc:824] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:07.184278 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1: Bootstrap starting.
I20260619 16:40:07.184651 22855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: No bootstrap required, opened a new log
I20260619 16:40:07.184546 22779 raft_consensus.cc:348] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.184755 22779 raft_consensus.cc:729] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aebb88973e8c437c932d88ccc0efc562, State: Initialized, Role: FOLLOWER
I20260619 16:40:07.185133 22779 consensus_queue.cc:260] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.185648 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.185935 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.186447 22779 sys_catalog.cc:564] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:07.186736 22704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:07.187402 22704 log.cc:824] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:07.189255 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:40:07.190289 22704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1: No bootstrap required, opened a new log
W20260619 16:40:07.191860 22876 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 16:40:07.190613 22678 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35194
W20260619 16:40:07.192075 22876 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:40:07.190789 22711 master_runner.cc:186] Error getting master registration for 127.0.98.62:41317: OK, instance_id { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" instance_seqno: 1781887206698021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:40:07.193936 22704 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.194113 22711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:40:07.194176 22704 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef761a3cf98c42339eb6ed53c7f61ed1, State: Initialized, Role: FOLLOWER
I20260619 16:40:07.194262 22855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.194502 22855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00b7eeac482b42feb0d179e1d7fcdc9b, State: Initialized, Role: FOLLOWER
I20260619 16:40:07.194500 22704 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.194835 22855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.195459 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.195730 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.195935 22704 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:07.196033 22678 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:40:07.196102 22855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:07.196147 22678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:07.197652 22678 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.198580 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.198655 22678 leader_election.cc:288] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [CANDIDATE]: Term 1 election: Requested vote from peers aebb88973e8c437c932d88ccc0efc562 (127.0.98.61:35863), 00b7eeac482b42feb0d179e1d7fcdc9b (127.0.98.60:45091)
I20260619 16:40:07.198699 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.199302 22753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aebb88973e8c437c932d88ccc0efc562"
I20260619 16:40:07.199539 22753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:07.199632 22829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b"
I20260619 16:40:07.199831 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:07.200855 22753 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef761a3cf98c42339eb6ed53c7f61ed1 in term 1.
I20260619 16:40:07.201162 22829 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef761a3cf98c42339eb6ed53c7f61ed1 in term 1.
I20260619 16:40:07.201399 22654 leader_election.cc:302] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aebb88973e8c437c932d88ccc0efc562, ef761a3cf98c42339eb6ed53c7f61ed1; no voters:
I20260619 16:40:07.202416 22878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:40:07.202629 22878 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [term 1 LEADER]: Becoming Leader. State: Replica: ef761a3cf98c42339eb6ed53c7f61ed1, State: Running, Role: LEADER
I20260619 16:40:07.202924 22878 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } }
I20260619 16:40:07.206816 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef761a3cf98c42339eb6ed53c7f61ed1. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.207373 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:07.209262 22785 master_runner.cc:263] No leader master found from master 00b7eeac482b42feb0d179e1d7fcdc9b
I20260619 16:40:07.209357 22785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 16:40:07.211697 22901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:40:07.211771 22901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 16:40:07.212080 22900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:40:07.212177 22900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:40:07.212478 22908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:40:07.213440 22908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:40:07.219307 22708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.207851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35208 (local address 127.0.98.62:41317)
0619 16:40:07.207959 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.207964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.209965 (+ 2001us) server_negotiation.cc:407] Connection header received
0619 16:40:07.210178 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.210181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.210229 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.210281 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.211187 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.211938 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.213334 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.213669 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.213748 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.215635 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.215639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.215642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.215671 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.217404 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.217909 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.217913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.217915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.218000 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.218318 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.218321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.218323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.218548 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.218651 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.219014 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.219152 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:40:07.229547 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.208480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.0.98.61:35863)
0619 16:40:07.208589 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.208594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.217612 (+ 9018us) server_negotiation.cc:407] Connection header received
0619 16:40:07.218320 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.218323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.218369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.218416 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.220187 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.220900 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.222152 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.222488 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.222565 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.224685 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.224689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.224692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.224720 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.226561 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.228162 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.228166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.228167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.228254 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.228541 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.228544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.228546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.228783 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.228887 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.229238 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.229383 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 16:40:07.231387 22829 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [term 1 FOLLOWER]: Refusing update from remote peer ef761a3cf98c42339eb6ed53c7f61ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:40:07.231663 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:07.209477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.0.98.60:45091)
0619 16:40:07.209608 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.209613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.217831 (+ 8218us) server_negotiation.cc:407] Connection header received
0619 16:40:07.218095 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.218100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.218145 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.218194 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.220219 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.221058 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.224277 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:07.224649 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:07.224720 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:07.227743 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:07.227747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:07.227750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:07.227780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:07.229514 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.230310 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.230314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.230315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.230404 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:07.230735 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:07.230740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:07.230742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:07.230953 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:07.231067 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:07.231390 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 16:40:07.231507 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 16:40:07.232331 22906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:40:07.233032 22752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [term 1 FOLLOWER]: Refusing update from remote peer ef761a3cf98c42339eb6ed53c7f61ed1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:40:07.236706 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef761a3cf98c42339eb6ed53c7f61ed1. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.236842 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.236953 22878 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:40:07.241003 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef761a3cf98c42339eb6ed53c7f61ed1. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.241134 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.243818 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.243935 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:07.244252 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.244354 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.244563 22914 mvcc.cc:205] Tried to move back new op lower bound from 7298610000784871424 to 7298610000706531328. Current Snapshot: MvccSnapshot[applied={T|T < 7298610000784871424}]
I20260619 16:40:07.245491 22915 mvcc.cc:205] Tried to move back new op lower bound from 7298610000784871424 to 7298610000706531328. Current Snapshot: MvccSnapshot[applied={T|T < 7298610000784871424}]
I20260619 16:40:07.248806 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.248924 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:07.249888 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef761a3cf98c42339eb6ed53c7f61ed1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41317 } } peers { permanent_uuid: "aebb88973e8c437c932d88ccc0efc562" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35863 } } peers { permanent_uuid: "00b7eeac482b42feb0d179e1d7fcdc9b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45091 } } }
I20260619 16:40:07.250000 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:07.250841 22917 mvcc.cc:205] Tried to move back new op lower bound from 7298610000784871424 to 7298610000706531328. Current Snapshot: MvccSnapshot[applied={T|T < 7298610000784871424}]
I20260619 16:40:07.257198 22908 catalog_manager.cc:1347] Generated new cluster ID: 43c27c230938463dbc0e9b6b43f966d5
I20260619 16:40:07.257270 22908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:40:07.292098 22908 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:40:07.293450 22908 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:40:07.311807 22908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef761a3cf98c42339eb6ed53c7f61ed1: Generated new TSK 0
I20260619 16:40:07.312693 22908 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:40:07.352345 22708 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.349415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.0.98.62:41317)
0619 16:40:07.349546 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.349554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.351265 (+ 1711us) server_negotiation.cc:407] Connection header received
0619 16:40:07.351452 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.351456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.351520 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.351575 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.352215 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35220: BlockingRecv error: recv got EOF from 127.0.0.1:35220 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 16:40:07.354019 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.350116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.98.61:35863)
0619 16:40:07.350229 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.350234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.352719 (+ 2485us) server_negotiation.cc:407] Connection header received
0619 16:40:07.353268 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.353272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.353350 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.353399 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.353911 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 16:40:07.354919 22861 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.350670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.0.98.60:45091)
0619 16:40:07.350777 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.350781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.353011 (+ 2230us) server_negotiation.cc:407] Connection header received
0619 16:40:07.354247 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.354250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.354300 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.354351 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.354814 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34414: BlockingRecv error: recv got EOF from 127.0.0.1:34414 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 16:40:07.358970 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.357137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46342 (local address 127.0.98.61:35863)
0619 16:40:07.357228 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.357232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.358118 (+ 886us) server_negotiation.cc:407] Connection header received
0619 16:40:07.358312 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.358315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.358378 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.358426 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.358881 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46342: BlockingRecv error: recv got EOF from 127.0.0.1:46342 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 16:40:07.363422 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.362114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.98.61:35863)
0619 16:40:07.362205 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.362209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.362672 (+ 463us) server_negotiation.cc:407] Connection header received
0619 16:40:07.362827 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.362830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.362870 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.362915 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.363340 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46356: BlockingRecv error: recv got EOF from 127.0.0.1:46356 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 16:40:07.368925 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.367450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.0.98.61:35863)
0619 16:40:07.367557 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.367562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.368043 (+ 481us) server_negotiation.cc:407] Connection header received
0619 16:40:07.368271 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.368275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.368321 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.368372 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.368838 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46368: BlockingRecv error: recv got EOF from 127.0.0.1:46368 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 16:40:07.376328 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.374853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.0.98.61:35863)
0619 16:40:07.374976 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.374980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.375494 (+ 514us) server_negotiation.cc:407] Connection header received
0619 16:40:07.375665 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.375668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.375712 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.375758 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.376230 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46378: BlockingRecv error: recv got EOF from 127.0.0.1:46378 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 16:40:07.388301 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.386511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.0.98.61:35863)
0619 16:40:07.386650 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.386655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.387250 (+ 595us) server_negotiation.cc:407] Connection header received
0619 16:40:07.387464 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.387467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.387517 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.387572 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.388198 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46392: BlockingRecv error: recv got EOF from 127.0.0.1:46392 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 16:40:07.408367 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.406610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.0.98.61:35863)
0619 16:40:07.406735 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.406740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.407368 (+ 628us) server_negotiation.cc:407] Connection header received
0619 16:40:07.407593 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.407597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.407653 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.407707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.408257 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46402: BlockingRecv error: recv got EOF from 127.0.0.1:46402 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 16:40:07.443670 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.442625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.0.98.61:35863)
0619 16:40:07.442794 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.442798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.442816 (+ 18us) server_negotiation.cc:407] Connection header received
0619 16:40:07.442959 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.442962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.443008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.443115 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.443578 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46416: BlockingRecv error: recv got EOF from 127.0.0.1:46416 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 16:40:07.511332 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.510271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.98.61:35863)
0619 16:40:07.510436 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.510440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.510473 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:40:07.510628 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.510630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.510676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.510744 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.511216 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46430: BlockingRecv error: recv got EOF from 127.0.0.1:46430 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 16:40:07.642414 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.641374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.98.61:35863)
0619 16:40:07.641584 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.641588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.641605 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:40:07.641712 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.641714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.641762 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.641833 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.642300 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46444: BlockingRecv error: recv got EOF from 127.0.0.1:46444 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260619 16:40:07.901396 22786 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:07.900394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.0.98.61:35863)
0619 16:40:07.900515 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:07.900518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:07.900533 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:40:07.900700 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:07.900703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:07.900747 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:07.900818 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:07.901243 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46448: BlockingRecv error: recv got EOF from 127.0.0.1:46448 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 16:40:08.193338 22876 catalog_manager.cc:1259] Loaded cluster ID: 43c27c230938463dbc0e9b6b43f966d5
I20260619 16:40:08.193415 22876 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: loading cluster ID for follower catalog manager: success
I20260619 16:40:08.194973 22876 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: acquiring CA information for follower catalog manager: success
I20260619 16:40:08.196313 22876 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aebb88973e8c437c932d88ccc0efc562: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:40:08.213200 22900 catalog_manager.cc:1259] Loaded cluster ID: 43c27c230938463dbc0e9b6b43f966d5
I20260619 16:40:08.213258 22900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: loading cluster ID for follower catalog manager: success
I20260619 16:40:08.214871 22900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: acquiring CA information for follower catalog manager: success
I20260619 16:40:08.215900 22900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00b7eeac482b42feb0d179e1d7fcdc9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:40:08.419309 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:08.415358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.0.98.61:35863)
0619 16:40:08.415574 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:08.415578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:08.415684 (+ 106us) server_negotiation.cc:407] Connection header received
0619 16:40:08.415814 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:08.415816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:08.415936 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:08.416040 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:08.416312 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.416859 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.417864 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.418107 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.418198 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:08.418485 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:08.418918 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:08.419010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:08.419135 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 16:40:08.419186 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260619 16:40:08.426165 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:08.422581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.0.98.60:45091)
0619 16:40:08.422805 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:08.422809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:08.422822 (+ 13us) server_negotiation.cc:407] Connection header received
0619 16:40:08.422931 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:08.422933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:08.423035 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:08.423159 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:08.423411 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.423931 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.424789 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.425025 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.425111 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:08.425451 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:08.425820 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:08.425913 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:08.425988 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 16:40:08.426037 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260619 16:40:08.433022 22987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:08.429285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.0.98.61:35863)
0619 16:40:08.429424 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:08.429428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:08.429667 (+ 239us) server_negotiation.cc:407] Connection header received
0619 16:40:08.429848 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:08.429851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:08.429897 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:08.429943 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:08.430318 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.431040 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.431928 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.432159 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.432272 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:08.432514 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:08.432588 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:08.432667 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:08.432798 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 16:40:08.432850 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 16:40:08.433642 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:08.429773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.0.98.60:45091)
0619 16:40:08.429880 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:08.429884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:08.430003 (+ 119us) server_negotiation.cc:407] Connection header received
0619 16:40:08.430187 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:08.430190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:08.430247 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:08.430325 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:08.430547 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.431104 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.432313 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.432660 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.432741 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:08.433130 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:08.433210 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:08.433285 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:08.433464 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 16:40:08.433511 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 16:40:08.433916 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:08.429035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35234 (local address 127.0.98.62:41317)
0619 16:40:08.429285 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:08.429289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:08.429306 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:40:08.429442 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:08.429445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:08.429597 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:08.429710 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:08.430059 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.430892 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.432313 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:08.432661 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:08.432759 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:08.433062 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:08.433418 (+ 356us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:08.433521 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:08.433677 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 16:40:08.433742 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":101,"threads_started":1}
I20260619 16:40:08.445591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22636
I20260619 16:40:08.451740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22711
I20260619 16:40:08.457422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 22785
2026-06-19T16:40:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3100 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 16:40:08 dist-test-slave-0f28 krb5kdc[23004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 16:40:08 dist-test-slave-0f28 krb5kdc[23004](info): set up 2 sockets
Jun 19 16:40:08 dist-test-slave-0f28 krb5kdc[23004](info): commencing operation
krb5kdc: starting...
W20260619 16:40:09.513738 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/test-user.keytab.
Jun 19 16:40:09 dist-test-slave-0f28 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T16:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T16:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 16:40:09.636274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43189,127.0.98.61:33835,127.0.98.60:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 16:40:09.741696 23017 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:40:09.742009 23017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:09.742061 23017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:09.742136 23017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:09.746264 23017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:09.746348 23017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:09.746387 23017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:09.746423 23017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:09.746457 23017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:09.751453 23017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43189,127.0.98.61:33835,127.0.98.60:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:09.752463 23017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:09.753468 23017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:09.761861 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:09.761860 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:09.761952 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:09.762142 23017 server_base.cc:1034] running on GCE node
I20260619 16:40:09.762633 23017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:09.763689 23017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:09.764883 23017 hybrid_clock.cc:648] HybridClock initialized: now 1781887209764871 us; error 36 us; skew 500 ppm
Jun 19 16:40:09 dist-test-slave-0f28 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:09.767690 23017 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 16:40:09.768563 23017 webserver.cc:460] Webserver started at http://127.0.98.62:33393/ using document root <none> and password file <none>
I20260619 16:40:09.769095 23017 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:09.769191 23017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:09.769475 23017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:09.771411 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data/instance:
uuid: "cae55accb9c643a9ba05da091c33b7f0"
format_stamp: "Formatted at 2026-06-19 16:40:09 on dist-test-slave-0f28"
server_key: "729078b37fd6cd86153c19c4eba581b5"
server_key_iv: "91fe4007ea1392cd8257a73ae5218095"
server_key_version: "encryptionkey@0"
I20260619 16:40:09.771998 23017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal/instance:
uuid: "cae55accb9c643a9ba05da091c33b7f0"
format_stamp: "Formatted at 2026-06-19 16:40:09 on dist-test-slave-0f28"
server_key: "729078b37fd6cd86153c19c4eba581b5"
server_key_iv: "91fe4007ea1392cd8257a73ae5218095"
server_key_version: "encryptionkey@0"
I20260619 16:40:09.775549 23017 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 16:40:09.778192 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:09.779398 23017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 16:40:09.779548 23017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal
uuid: "cae55accb9c643a9ba05da091c33b7f0"
format_stamp: "Formatted at 2026-06-19 16:40:09 on dist-test-slave-0f28"
server_key: "729078b37fd6cd86153c19c4eba581b5"
server_key_iv: "91fe4007ea1392cd8257a73ae5218095"
server_key_version: "encryptionkey@0"
I20260619 16:40:09.779680 23017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:09.807663 23017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:09.810559 23017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:09.810773 23017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:09.817548 23017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43189
I20260619 16:40:09.817870 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43189 every 8 connection(s)
I20260619 16:40:09.818506 23017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/data/info.pb
I20260619 16:40:09.821187 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:09.821836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 23017
I20260619 16:40:09.821965 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-0/wal/instance
I20260619 16:40:09.822243 392 external_mini_cluster.cc:1428] Setting key 58ba529955fce7ac3f1633eec18fab9f
I20260619 16:40:09.824867 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 16:40:09 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:09.840307 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.827105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.0.98.62:43189)
0619 16:40:09.827501 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:09.827520 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:09.827799 (+ 279us) server_negotiation.cc:407] Connection header received
0619 16:40:09.828787 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:09.828792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:09.828970 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:09.829159 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:09.830014 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:09.830954 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:09.832034 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:09.832492 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:09.832657 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:09.835290 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:09.835295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:09.835304 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:09.835325 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:09.837014 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:09.837809 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:09.837813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:09.837815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:09.837918 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:09.838385 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:09.838388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:09.838389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:09.838742 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:09.838847 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:09.839337 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 16:40:09.839517 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":140,"threads_started":1}
I20260619 16:40:09.840312 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.827106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43189 (local address 127.0.0.1:47338)
0619 16:40:09.827659 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.827695 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:09.828483 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:09.829146 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:09.829151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:09.829586 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:09.829869 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:09.829883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:09.831084 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:09.831093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:09.831875 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:09.831884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:09.832645 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:09.832649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:09.832758 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:09.833439 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:09.833453 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:09.835141 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:09.837259 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:09.837279 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:09.837290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:09.837659 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:09.838028 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:09.838033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:09.838035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:09.838239 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:09.838871 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:09.838880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:09.839068 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 16:40:09.839260 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 16:40:09.839652 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":78,"spinlock_wait_cycles":29568,"thread_start_us":156,"threads_started":1}
I20260619 16:40:09.842679 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:09.843353 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.843129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60874)
0619 16:40:09.843239 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.843303 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 16:40:09.843587 23036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
I20260619 16:40:09.844512 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.844263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60890)
0619 16:40:09.844391 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.844450 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 16:40:09.844682 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33835: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 16:40:09.849092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33835
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43189,127.0.98.61:33835,127.0.98.60:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:40:09.893991 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } attempt: 1
I20260619 16:40:09.894944 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.894606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60900)
0619 16:40:09.894781 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.894854 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 16:40:09.895766 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.895539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60912)
0619 16:40:09.895649 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.895702 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 16:40:09.895944 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33835: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
W20260619 16:40:09.956532 23092 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:40:09.956842 23092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:09.956915 23092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:09.956993 23092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:09.960891 23092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:09.960978 23092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:09.961014 23092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:09.961050 23092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:09.961081 23092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:09.964133 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } attempt: 2
I20260619 16:40:09.965049 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.964723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60924)
0619 16:40:09.964884 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.964953 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 16:40:09.965845 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:09.965629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60934)
0619 16:40:09.965734 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:09.965784 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 16:40:09.966015 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33835: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33835: connect: Connection refused (error 111)
I20260619 16:40:09.966679 23092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43189,127.0.98.61:33835,127.0.98.60:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:09.967618 23092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:09.968546 23092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 16:40:09.976773 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:09.976780 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:09.976922 23092 server_base.cc:1034] running on GCE node
W20260619 16:40:09.976789 23100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:09.977644 23092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:09.978750 23092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:09.979910 23092 hybrid_clock.cc:648] HybridClock initialized: now 1781887209979891 us; error 30 us; skew 500 ppm
Jun 19 16:40:09 dist-test-slave-0f28 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:09.982672 23092 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 16:40:09.983702 23092 webserver.cc:460] Webserver started at http://127.0.98.61:34411/ using document root <none> and password file <none>
I20260619 16:40:09.984346 23092 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:09.984433 23092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:09.984637 23092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:09.986649 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data/instance:
uuid: "e94790ea989c4be3802c6b54f551cca5"
format_stamp: "Formatted at 2026-06-19 16:40:09 on dist-test-slave-0f28"
server_key: "5654e6b36805813fe965acb907e1352f"
server_key_iv: "9cf3523787e11a4d10d208f534d4dfcf"
server_key_version: "encryptionkey@0"
I20260619 16:40:09.987210 23092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal/instance:
uuid: "e94790ea989c4be3802c6b54f551cca5"
format_stamp: "Formatted at 2026-06-19 16:40:09 on dist-test-slave-0f28"
server_key: "5654e6b36805813fe965acb907e1352f"
server_key_iv: "9cf3523787e11a4d10d208f534d4dfcf"
server_key_version: "encryptionkey@0"
I20260619 16:40:09.990728 23092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 16:40:09.993085 23107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:09.994324 23092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:40:09.994454 23092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal
uuid: "e94790ea989c4be3802c6b54f551cca5"
format_stamp: "Formatted at 2026-06-19 16:40:09 on dist-test-slave-0f28"
server_key: "5654e6b36805813fe965acb907e1352f"
server_key_iv: "9cf3523787e11a4d10d208f534d4dfcf"
server_key_version: "encryptionkey@0"
I20260619 16:40:09.994561 23092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:10.010143 23092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:10.013200 23092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:10.013422 23092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:10.020330 23092 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33835
I20260619 16:40:10.020627 23159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33835 every 8 connection(s)
I20260619 16:40:10.021258 23092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/data/info.pb
I20260619 16:40:10.023851 23160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:10.024989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 23092
I20260619 16:40:10.025089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-1/wal/instance
I20260619 16:40:10.025465 392 external_mini_cluster.cc:1428] Setting key 7c7ecc99422fab15c34f86932dcb1f05
I20260619 16:40:10.027510 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:10.042610 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.029307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.0.98.62:43189)
0619 16:40:10.029463 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.029468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.030396 (+ 928us) server_negotiation.cc:407] Connection header received
0619 16:40:10.031706 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.031709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.031759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.031832 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.033424 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.033999 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.035178 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.035402 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.035483 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.038425 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.038428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.038430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.038452 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.039937 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.040458 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.040462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.040464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.040543 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.040889 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.040892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.040894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.041068 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.041174 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.042333 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.042455 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:40:10.042999 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.029682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43189 (local address 127.0.0.1:47342)
0619 16:40:10.030203 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.030264 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.031463 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.032069 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.032080 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.032550 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.033252 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.033274 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.034128 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.034131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.035063 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.035073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.035591 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.035595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.035770 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.036376 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.036400 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.038254 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.040071 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.040077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.040090 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.040348 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.040645 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.040649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.040651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.040785 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.041183 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.041192 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.041493 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.041671 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.042012 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":108,"threads_started":1}
I20260619 16:40:10.044322 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 16:40:10.052817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFXRShK/build/debug/bin/kudu
/tmp/dist-test-taskFXRShK/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36865
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43189,127.0.98.61:33835,127.0.98.60:36865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 16:40:10.054054 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.044912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60936)
0619 16:40:10.045102 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.045120 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.045200 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.045735 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.045741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.045923 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.046026 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.046033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.047015 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.047019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.047667 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.047698 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.048299 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.048304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.048420 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.048800 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.048811 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.050073 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.052076 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.052078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.052080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.052343 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.052682 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.052685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.052687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.052777 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.053382 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.053388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.053500 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.053629 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.053778 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 16:40:10.054102 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.045028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.0.98.61:33835)
0619 16:40:10.045311 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.045360 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.045383 (+ 23us) server_negotiation.cc:407] Connection header received
0619 16:40:10.045426 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.045430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.045592 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.045708 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.046140 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.046905 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.047808 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.048180 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.048261 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.050201 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.050209 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.050218 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.050238 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.051965 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.052488 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.052493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.052494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.052577 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.052899 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.052902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.052903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.053242 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.053419 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.053733 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.053854 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":129,"threads_started":1}
I20260619 16:40:10.056324 23160 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:10.057075 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.056832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36162)
0619 16:40:10.056942 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.057012 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 16:40:10.057206 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } attempt: 3
W20260619 16:40:10.057446 23110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
I20260619 16:40:10.059207 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.058952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36166)
0619 16:40:10.059075 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.059148 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:40:10.059461 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:40:10.066708 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.057671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60946)
0619 16:40:10.057790 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.057806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.057890 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.058272 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.058279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.058492 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.058659 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.058671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.059622 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.059628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.060631 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.060645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.061111 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.061115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.061262 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.061719 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.061740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.063075 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.065171 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.065174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.065176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.065493 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.065778 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.065780 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.065782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.065858 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.066276 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.066278 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.066363 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.066467 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.066591 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 16:40:10.066792 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.057926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.0.98.61:33835)
0619 16:40:10.058025 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.058029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.058041 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:40:10.058076 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.058078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.058122 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.058201 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.058798 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.059468 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.060762 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.060985 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.061061 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.063221 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.063227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.063233 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.063265 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.065035 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.065594 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.065597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.065599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.065685 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.065943 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.065946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.065948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.066146 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.066241 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.066537 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.066654 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 16:40:10.068125 23085 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:10.068851 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.068604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36170)
0619 16:40:10.068738 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.068789 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 16:40:10.069600 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.069379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36172)
0619 16:40:10.069503 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.069547 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 16:40:10.069782 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
I20260619 16:40:10.100955 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } attempt: 1
I20260619 16:40:10.101804 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.101508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36182)
0619 16:40:10.101658 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.101725 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 16:40:10.102571 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.102377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36184)
0619 16:40:10.102476 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.102520 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 16:40:10.102773 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
I20260619 16:40:10.108767 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } attempt: 1
I20260619 16:40:10.109525 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.109227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36188)
0619 16:40:10.109410 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.109461 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 16:40:10.110242 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.110040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36198)
0619 16:40:10.110157 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.110199 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 16:40:10.110421 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
W20260619 16:40:10.165639 23166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 16:40:10.165882 23166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 16:40:10.165905 23166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 16:40:10.166002 23166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 16:40:10.169970 23166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 16:40:10.170019 23166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 16:40:10.170038 23166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 16:40:10.170106 23166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 16:40:10.170130 23166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 16:40:10.175238 23166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43189,127.0.98.61:33835,127.0.98.60:36865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 15:59:16 UTC on ccycloud.quasar-ddinlc.root.comops.site
build id 4456153
I20260619 16:40:10.176254 23166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 16:40:10.177250 23166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 16:40:10.177944 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } attempt: 2
I20260619 16:40:10.178604 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } attempt: 2
I20260619 16:40:10.178879 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.178512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36212)
0619 16:40:10.178706 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.178781 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 16:40:10.179279 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.178977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36220)
0619 16:40:10.179125 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.179201 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 16:40:10.179929 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.179659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36234)
0619 16:40:10.179808 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.179861 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 16:40:10.180104 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.179847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36246)
0619 16:40:10.179972 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.180031 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 16:40:10.180255 23160 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
W20260619 16:40:10.180305 23085 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36865: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36865: connect: Connection refused (error 111)
W20260619 16:40:10.186432 23175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:10.186479 23172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 16:40:10.186560 23173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 16:40:10.186661 23166 server_base.cc:1034] running on GCE node
I20260619 16:40:10.187247 23166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 16:40:10.188355 23166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 16:40:10.189565 23166 hybrid_clock.cc:648] HybridClock initialized: now 1781887210189534 us; error 42 us; skew 500 ppm
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 16:40:10.192255 23166 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 16:40:10.193043 23166 webserver.cc:460] Webserver started at http://127.0.98.60:33001/ using document root <none> and password file <none>
I20260619 16:40:10.193609 23166 fs_manager.cc:359] Metadata directory not provided
I20260619 16:40:10.193671 23166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 16:40:10.193971 23166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 16:40:10.196125 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data/instance:
uuid: "0c6e0320184545b397c71ae4ea8c3491"
format_stamp: "Formatted at 2026-06-19 16:40:10 on dist-test-slave-0f28"
server_key: "d895d414ebae32d40a6f069d8c2234d8"
server_key_iv: "ea7b306578e51c1d3cb6d0eecb439140"
server_key_version: "encryptionkey@0"
I20260619 16:40:10.196673 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal/instance:
uuid: "0c6e0320184545b397c71ae4ea8c3491"
format_stamp: "Formatted at 2026-06-19 16:40:10 on dist-test-slave-0f28"
server_key: "d895d414ebae32d40a6f069d8c2234d8"
server_key_iv: "ea7b306578e51c1d3cb6d0eecb439140"
server_key_version: "encryptionkey@0"
I20260619 16:40:10.200744 23166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 16:40:10.203284 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 16:40:10.204640 23166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 16:40:10.204746 23166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal
uuid: "0c6e0320184545b397c71ae4ea8c3491"
format_stamp: "Formatted at 2026-06-19 16:40:10 on dist-test-slave-0f28"
server_key: "d895d414ebae32d40a6f069d8c2234d8"
server_key_iv: "ea7b306578e51c1d3cb6d0eecb439140"
server_key_version: "encryptionkey@0"
I20260619 16:40:10.204833 23166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 16:40:10.226416 23166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 16:40:10.229809 23166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 16:40:10.229954 23166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 16:40:10.236649 23166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36865
I20260619 16:40:10.236985 23234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36865 every 8 connection(s)
I20260619 16:40:10.237618 23166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/data/info.pb
I20260619 16:40:10.240265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFXRShK/build/debug/bin/kudu as pid 23166
I20260619 16:40:10.240391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFXRShK/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781886927079793-392-0/minicluster-data/master-2/wal/instance
I20260619 16:40:10.240633 23235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 16:40:10.240664 392 external_mini_cluster.cc:1428] Setting key f2bffe3ec18418fe20452cb7a6081ef2
I20260619 16:40:10.244902 23235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:10.255522 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.241961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.0.98.60:36865)
0619 16:40:10.242368 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.242385 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.242418 (+ 33us) server_negotiation.cc:407] Connection header received
0619 16:40:10.243263 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.243279 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.243596 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.243912 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.244686 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.245819 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.246949 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.247693 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.247841 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.250513 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.250519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.250527 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.250551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.252268 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.252715 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.252724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.252732 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.252856 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.253194 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.253198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.253200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.253741 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.253943 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.254223 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.254560 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260619 16:40:10.255967 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.246780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.0.98.62:43189)
0619 16:40:10.246920 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.246927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.247299 (+ 372us) server_negotiation.cc:407] Connection header received
0619 16:40:10.247412 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.247414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.247456 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.247522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.248274 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.248778 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.249896 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.250170 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.250251 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.252504 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.252507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.252510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.252533 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.254147 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.254688 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.254691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.254693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.254777 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.255082 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.255085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.255087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.255286 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.255382 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.255746 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.255865 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:40:10.256018 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.246664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43189 (local address 127.0.0.1:47344)
0619 16:40:10.247155 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.247197 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.247320 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.247594 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.247600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.248035 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.248152 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.248157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.248913 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.248918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.249781 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.249790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.250320 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.250328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.250453 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.250961 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.250973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.252389 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.254263 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.254282 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.254288 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.254589 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.254895 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.254898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.254899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.254980 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.255388 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.255393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.255480 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.255656 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.255827 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":126,"threads_started":1}
I20260619 16:40:10.257164 23235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 16:40:10.266108 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.257648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33835 (local address 127.0.0.1:60962)
0619 16:40:10.258053 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.258067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.258149 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.258381 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.258385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.258557 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.258636 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.258641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.259501 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.259504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.260125 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.260134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.260689 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.260692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.260784 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.261230 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.261241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.262560 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.264506 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.264512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.264516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.264777 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.265096 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.265098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.265100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.265174 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.265632 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.265635 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.265731 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.265848 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.265984 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343}
I20260619 16:40:10.266206 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.257783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.0.98.61:33835)
0619 16:40:10.257954 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.257959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.258143 (+ 184us) server_negotiation.cc:407] Connection header received
0619 16:40:10.258233 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.258235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.258277 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.258336 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.258760 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.259375 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.260238 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.260583 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.260677 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.262697 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.262703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.262708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.262741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.264376 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.264871 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.264876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.264878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.264967 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.265269 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.265275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.265277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.265520 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.265621 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.265896 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.266031 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 16:40:10.266921 23235 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } has no permanent_uuid. Determining permanent_uuid...
I20260619 16:40:10.267325 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.258086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.0.98.62:43189)
0619 16:40:10.258203 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.258208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.258222 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:40:10.258253 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.258256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.258299 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.258391 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.259118 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.259693 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.260601 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.260947 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.261041 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.263761 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.263765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.263767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.263796 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.265499 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.265961 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.265964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.265966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.266087 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.266426 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.266432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.266435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.266669 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.266907 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.267029 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.267147 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 16:40:10.271454 23160 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } attempt: 3
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 16:40:10.275978 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.267545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.0.98.60:36865)
0619 16:40:10.267630 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.267646 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.267658 (+ 12us) server_negotiation.cc:407] Connection header received
0619 16:40:10.267723 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.267728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.267764 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.267829 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.268184 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.268918 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.269926 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.270238 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.270339 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.272701 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.272704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.272711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.272730 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.274288 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.274850 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.274853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.274854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.274920 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.275207 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.275210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.275212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.275363 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.275465 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.275686 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.275819 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 16:40:10.276007 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.267360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36266)
0619 16:40:10.267521 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.267543 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.267647 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.267917 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.267920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.268042 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.268094 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.268098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.269075 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.269077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.269815 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.269825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.270345 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.270347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.270403 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.270868 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.270882 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.272571 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.274400 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.274402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.274404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.274570 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.275035 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.275038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.275039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.275112 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.275455 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.275458 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.275536 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.275641 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.275875 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":116}
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:40:10.278079 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.268345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.0.98.61:33835)
0619 16:40:10.268444 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.268448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.268465 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:40:10.268550 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.268552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.268583 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.268642 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.269295 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.270013 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.270808 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.271014 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.271095 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.273875 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.273878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.273881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.273903 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.275503 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.275957 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.275960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.275962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.276046 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.276768 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.276774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.276779 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.277002 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.277208 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.277885 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.277972 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40,"spinlock_wait_cycles":1792}
I20260619 16:40:10.279469 23085 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } attempt: 3
I20260619 16:40:10.279640 23235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: Bootstrap starting.
I20260619 16:40:10.281776 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.272212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36272)
0619 16:40:10.272303 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.272319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.272387 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.272898 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.272904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.273096 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.273203 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.273210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.274366 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.274370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.275213 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.275226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.275895 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.275902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.275988 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.276388 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.276403 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.277739 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.280151 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.280155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.280157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.280396 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.280713 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.280717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.280719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.280802 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.281286 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.281288 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.281400 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.281483 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.281599 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"spinlock_wait_cycles":2304}
I20260619 16:40:10.281802 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 16:40:10.282388 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.272310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.0.98.60:36865)
0619 16:40:10.272647 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.272658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.272674 (+ 16us) server_negotiation.cc:407] Connection header received
0619 16:40:10.272709 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.272712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.272764 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.272856 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.273431 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.274208 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.275371 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.275764 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.276196 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.277853 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.277856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.277859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.277881 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.280015 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.280518 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.280521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.280523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.280605 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.280916 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.280921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.280925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.281169 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.282011 (+ 842us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.282081 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.282190 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260619 16:40:10.282776 23235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:10.283779 23235 log.cc:824] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:10.282989 23059 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cae55accb9c643a9ba05da091c33b7f0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47350
I20260619 16:40:10.285187 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: Bootstrap starting.
Jun 19 16:40:10 dist-test-slave-0f28 krb5kdc[23004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781887209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 16:40:10.286123 23235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: No bootstrap required, opened a new log
I20260619 16:40:10.287757 23160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:10.288558 23160 log.cc:824] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:10.290125 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.279852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36865 (local address 127.0.0.1:36282)
0619 16:40:10.280003 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 16:40:10.280019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 16:40:10.280103 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 16:40:10.280714 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 16:40:10.280717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 16:40:10.280910 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 16:40:10.281039 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.281048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.282362 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.282366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.283236 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 16:40:10.283248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.284259 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.284265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 16:40:10.284353 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.284856 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 16:40:10.284872 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 16:40:10.286364 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 16:40:10.288379 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.288383 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.288397 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.288724 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.289051 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 16:40:10.289054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 16:40:10.289056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 16:40:10.289145 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 16:40:10.289621 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 16:40:10.289626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 16:40:10.289729 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 16:40:10.289841 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 16:40:10.289970 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 16:40:10.290202 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.280302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.0.98.60:36865)
0619 16:40:10.280436 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.280440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.280454 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:40:10.280488 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.280491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.280530 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.280605 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.281184 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.281993 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.283622 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.283977 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.284083 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.286574 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.286579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.286582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.286607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.288265 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.288853 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.288857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.288859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.288948 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.289250 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.289253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.289255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.289484 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.289573 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.289928 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.290057 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 16:40:10.290661 23235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.290951 23235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c6e0320184545b397c71ae4ea8c3491, State: Initialized, Role: FOLLOWER
I20260619 16:40:10.291313 23235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.291536 23160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: No bootstrap required, opened a new log
I20260619 16:40:10.292771 23235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:10.293365 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: Bootstrap starting.
I20260619 16:40:10.295480 23160 raft_consensus.cc:348] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.295688 23160 raft_consensus.cc:729] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94790ea989c4be3802c6b54f551cca5, State: Initialized, Role: FOLLOWER
I20260619 16:40:10.295997 23160 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.296231 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: Neither blocks nor log segments found. Creating new log.
I20260619 16:40:10.297183 23085 log.cc:824] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: Log is configured to *not* fsync() on all Append() calls
I20260619 16:40:10.297297 23160 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 16:40:10.300307 23254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:40:10.300498 23254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:40:10.296577 23166 master_runner.cc:186] Error getting master registration for 127.0.98.62:43189: OK, instance_id { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" instance_seqno: 1781887209815654 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:40:10.300851 23166 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:40:10.301036 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.301383 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.301430 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.301502 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: No bootstrap required, opened a new log
I20260619 16:40:10.301517 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.303817 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.303995 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae55accb9c643a9ba05da091c33b7f0, State: Initialized, Role: FOLLOWER
I20260619 16:40:10.304237 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.300640 23092 master_runner.cc:186] Error getting master registration for 127.0.98.62:43189: OK, instance_id { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" instance_seqno: 1781887209815654 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 16:40:10.304659 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.304738 23092 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 16:40:10.304792 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.305516 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 16:40:10.305612 23059 raft_consensus.cc:482] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 16:40:10.305732 23059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 0 FOLLOWER]: Advancing to term 1
W20260619 16:40:10.306317 23268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:40:10.306373 23268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:40:10.306921 23059 raft_consensus.cc:504] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.307829 23059 leader_election.cc:288] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [CANDIDATE]: Term 1 election: Requested vote from peers e94790ea989c4be3802c6b54f551cca5 (127.0.98.61:33835), 0c6e0320184545b397c71ae4ea8c3491 (127.0.98.60:36865)
I20260619 16:40:10.308681 23134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cae55accb9c643a9ba05da091c33b7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e94790ea989c4be3802c6b54f551cca5"
I20260619 16:40:10.308878 23134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 16:40:10.310142 23134 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae55accb9c643a9ba05da091c33b7f0 in term 1.
I20260619 16:40:10.310539 23209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cae55accb9c643a9ba05da091c33b7f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0c6e0320184545b397c71ae4ea8c3491"
I20260619 16:40:10.310747 23209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [term 0 FOLLOWER]: Advancing to term 1
W20260619 16:40:10.312263 23038 tablet.cc:2369] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 16:40:10.312464 23209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cae55accb9c643a9ba05da091c33b7f0 in term 1.
W20260619 16:40:10.312885 23281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 16:40:10.312958 23281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 16:40:10.313112 23035 leader_election.cc:302] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c6e0320184545b397c71ae4ea8c3491, cae55accb9c643a9ba05da091c33b7f0; no voters:
I20260619 16:40:10.313423 23269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 16:40:10.313642 23269 raft_consensus.cc:686] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [term 1 LEADER]: Becoming Leader. State: Replica: cae55accb9c643a9ba05da091c33b7f0, State: Running, Role: LEADER
I20260619 16:40:10.313975 23269 consensus_queue.cc:237] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } }
I20260619 16:40:10.315763 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae55accb9c643a9ba05da091c33b7f0. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.315876 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:10.323102 23289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 16:40:10.324071 23289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 16:40:10.331018 23209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [term 1 FOLLOWER]: Refusing update from remote peer cae55accb9c643a9ba05da091c33b7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:40:10.331713 23283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 16:40:10.336673 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae55accb9c643a9ba05da091c33b7f0. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.336803 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.337867 23134 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [term 1 FOLLOWER]: Refusing update from remote peer cae55accb9c643a9ba05da091c33b7f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 16:40:10.338433 23283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 16:40:10.342253 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cae55accb9c643a9ba05da091c33b7f0. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.342393 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.344576 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.322686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.0.98.60:36865)
0619 16:40:10.322790 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.322795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.331666 (+ 8871us) server_negotiation.cc:407] Connection header received
0619 16:40:10.332608 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.332612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.332670 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.332717 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.333952 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.334776 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.338987 (+ 4211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.339329 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.339402 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.341088 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.341095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.341098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.341124 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.342807 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.343279 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.343283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.343285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.343368 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.343664 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.343667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.343669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.343870 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.343973 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.344310 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.344426 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 16:40:10.345499 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.321789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.0.98.62:43189)
0619 16:40:10.321905 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.321909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.329590 (+ 7681us) server_negotiation.cc:407] Connection header received
0619 16:40:10.329785 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.329789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.329835 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.329886 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.330900 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.331648 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.332858 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.333251 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.333354 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.335180 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.335184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.335187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.335212 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.336874 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.337367 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.337371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.337373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.337454 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.339185 (+ 1731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.339189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.339191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.339404 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.339513 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.345089 (+ 5576us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.345354 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:40:10.348505 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.348655 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:10.349720 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.349841 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.350188 23293 mvcc.cc:205] Tried to move back new op lower bound from 7298610013508247552 to 7298610013449424896. Current Snapshot: MvccSnapshot[applied={T|T < 7298610013508247552}]
I20260619 16:40:10.351320 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:10.322192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.0.98.61:33835)
0619 16:40:10.322338 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.322342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.331384 (+ 9042us) server_negotiation.cc:407] Connection header received
0619 16:40:10.331752 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.331757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.331812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.331862 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.334923 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.335716 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.337956 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:10.338292 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:10.338359 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:10.346761 (+ 8402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 16:40:10.346765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 16:40:10.346767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 16:40:10.346794 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 16:40:10.348439 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.349978 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.349981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.349983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.350069 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 16:40:10.350415 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 16:40:10.350419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 16:40:10.350421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 16:40:10.350621 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 16:40:10.350740 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:10.351068 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 16:40:10.351180 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 16:40:10.352445 23296 mvcc.cc:205] Tried to move back new op lower bound from 7298610013508247552 to 7298610013449424896. Current Snapshot: MvccSnapshot[applied={T|T < 7298610013508247552}]
I20260619 16:40:10.353125 23269 sys_catalog.cc:455] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.353238 23269 sys_catalog.cc:458] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0 [sys.catalog]: This master's current role is: LEADER
I20260619 16:40:10.354139 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cae55accb9c643a9ba05da091c33b7f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae55accb9c643a9ba05da091c33b7f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43189 } } peers { permanent_uuid: "e94790ea989c4be3802c6b54f551cca5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33835 } } peers { permanent_uuid: "0c6e0320184545b397c71ae4ea8c3491" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36865 } } }
I20260619 16:40:10.354257 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 16:40:10.356597 23294 mvcc.cc:205] Tried to move back new op lower bound from 7298610013508247552 to 7298610013449424896. Current Snapshot: MvccSnapshot[applied={T|T < 7298610013508247552}]
I20260619 16:40:10.358494 23289 catalog_manager.cc:1347] Generated new cluster ID: 9d9eef34f17e4d3f851785be9b15e8d3
I20260619 16:40:10.358568 23289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 16:40:10.387607 23289 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 16:40:10.389034 23289 catalog_manager.cc:1504] Loading token signing keys...
I20260619 16:40:10.406670 23289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cae55accb9c643a9ba05da091c33b7f0: Generated new TSK 0
I20260619 16:40:10.407620 23289 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 16:40:10.438127 23088 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.434866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.0.98.62:43189)
0619 16:40:10.434989 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.434994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.436737 (+ 1743us) server_negotiation.cc:407] Connection header received
0619 16:40:10.436934 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.436937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.437109 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.437197 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.437965 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47366: BlockingRecv error: recv got EOF from 127.0.0.1:47366 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 16:40:10.440052 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.435567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.0.98.61:33835)
0619 16:40:10.435680 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.435684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.438578 (+ 2894us) server_negotiation.cc:407] Connection header received
0619 16:40:10.439212 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.439216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.439272 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.439329 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.439916 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60988: BlockingRecv error: recv got EOF from 127.0.0.1:60988 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 16:40:10.441059 23238 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.436187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.98.60:36865)
0619 16:40:10.436319 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.436324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.438926 (+ 2602us) server_negotiation.cc:407] Connection header received
0619 16:40:10.440314 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.440318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.440374 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.440447 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.440942 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36292: BlockingRecv error: recv got EOF from 127.0.0.1:36292 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:40:10.446427 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.444328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.0.98.61:33835)
0619 16:40:10.444452 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.444456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.444919 (+ 463us) server_negotiation.cc:407] Connection header received
0619 16:40:10.445156 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.445160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.445240 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.445299 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.446188 (+ 889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60992: BlockingRecv error: recv got EOF from 127.0.0.1:60992 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 16:40:10.451506 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.449970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.0.98.61:33835)
0619 16:40:10.450078 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.450082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.450620 (+ 538us) server_negotiation.cc:407] Connection header received
0619 16:40:10.450804 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.450807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.450856 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.450908 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.451408 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60994: BlockingRecv error: recv got EOF from 127.0.0.1:60994 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:40:10.456879 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.455412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.0.98.61:33835)
0619 16:40:10.455515 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.455519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.456016 (+ 497us) server_negotiation.cc:407] Connection header received
0619 16:40:10.456207 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.456211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.456258 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.456309 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.456786 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60998: BlockingRecv error: recv got EOF from 127.0.0.1:60998 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 16:40:10.464643 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.462979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32780 (local address 127.0.98.61:33835)
0619 16:40:10.463115 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.463119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.463687 (+ 568us) server_negotiation.cc:407] Connection header received
0619 16:40:10.463893 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.463897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.463950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.464004 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.464535 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32780: BlockingRecv error: recv got EOF from 127.0.0.1:32780 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 16:40:10.477416 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.475297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.98.61:33835)
0619 16:40:10.475432 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.475436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.476112 (+ 676us) server_negotiation.cc:407] Connection header received
0619 16:40:10.476350 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.476353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.476417 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.476494 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.477251 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32782: BlockingRecv error: recv got EOF from 127.0.0.1:32782 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 16:40:10.497717 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.495759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.0.98.61:33835)
0619 16:40:10.495899 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.495906 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.496585 (+ 679us) server_negotiation.cc:407] Connection header received
0619 16:40:10.496787 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.496791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.496846 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.496905 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.497603 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32794: BlockingRecv error: recv got EOF from 127.0.0.1:32794 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 16:40:10.533645 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.532216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.0.98.61:33835)
0619 16:40:10.532357 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.532362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.532655 (+ 293us) server_negotiation.cc:407] Connection header received
0619 16:40:10.532841 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.532847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.532907 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.533022 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.533517 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32804: BlockingRecv error: recv got EOF from 127.0.0.1:32804 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 16:40:10.601431 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.600098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.0.98.61:33835)
0619 16:40:10.600280 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.600285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.600415 (+ 130us) server_negotiation.cc:407] Connection header received
0619 16:40:10.600592 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.600596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.600661 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.600770 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.601268 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32820: BlockingRecv error: recv got EOF from 127.0.0.1:32820 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260619 16:40:10.732887 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.731625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.0.98.61:33835)
0619 16:40:10.731779 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.731783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.731980 (+ 197us) server_negotiation.cc:407] Connection header received
0619 16:40:10.732197 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.732200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.732254 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.732329 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.732788 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32828: BlockingRecv error: recv got EOF from 127.0.0.1:32828 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 16:40:10.992220 23165 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 16:40:10.991139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.0.98.61:33835)
0619 16:40:10.991293 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:10.991298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:10.991315 (+ 17us) server_negotiation.cc:407] Connection header received
0619 16:40:10.991468 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:10.991471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:10.991516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:10.991610 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 16:40:10.992073 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32840: BlockingRecv error: recv got EOF from 127.0.0.1:32840 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 16:40:11.301911 23254 catalog_manager.cc:1259] Loaded cluster ID: 9d9eef34f17e4d3f851785be9b15e8d3
I20260619 16:40:11.302043 23254 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: loading cluster ID for follower catalog manager: success
I20260619 16:40:11.303668 23254 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: acquiring CA information for follower catalog manager: success
I20260619 16:40:11.304782 23254 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0c6e0320184545b397c71ae4ea8c3491: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:40:11.307528 23268 catalog_manager.cc:1259] Loaded cluster ID: 9d9eef34f17e4d3f851785be9b15e8d3
I20260619 16:40:11.307610 23268 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: loading cluster ID for follower catalog manager: success
I20260619 16:40:11.309190 23268 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: acquiring CA information for follower catalog manager: success
I20260619 16:40:11.310245 23268 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e94790ea989c4be3802c6b54f551cca5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 16:40:11.510764 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:11.506425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.0.98.61:33835)
0619 16:40:11.506728 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:11.506732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:11.506784 (+ 52us) server_negotiation.cc:407] Connection header received
0619 16:40:11.506914 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:11.506916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:11.507057 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:11.507160 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:11.507483 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.508078 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.509108 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.509431 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.509570 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:11.509929 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:11.510351 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:11.510498 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:11.510544 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 16:40:11.510600 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
I20260619 16:40:11.517081 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:11.513657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.98.60:36865)
0619 16:40:11.513850 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:11.513853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:11.513867 (+ 14us) server_negotiation.cc:407] Connection header received
0619 16:40:11.513954 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:11.513956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:11.514061 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:11.514149 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:11.514318 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.514814 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.515611 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.515831 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.515902 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:11.516202 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:11.516615 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:11.516721 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:11.516837 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 16:40:11.516884 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260619 16:40:11.524452 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:11.520326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.0.98.61:33835)
0619 16:40:11.520425 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:11.520429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:11.520785 (+ 356us) server_negotiation.cc:407] Connection header received
0619 16:40:11.520865 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:11.520868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:11.520921 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:11.521005 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:11.521322 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.522093 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.523095 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.523437 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.523533 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:11.523913 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:11.523990 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:11.524059 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:11.524244 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 16:40:11.524302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 16:40:11.524670 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:11.520034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.0.98.62:43189)
0619 16:40:11.520396 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:11.520400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:11.520415 (+ 15us) server_negotiation.cc:407] Connection header received
0619 16:40:11.520563 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:11.520566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:11.520725 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:11.521100 (+ 375us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:11.521251 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.521818 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.523089 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.523377 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.523488 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:11.523775 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:11.524095 (+ 320us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:11.524395 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:11.524432 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 16:40:11.524485 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":77,"threads_started":1}
I20260619 16:40:11.524668 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 16:40:11.520179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.0.98.60:36865)
0619 16:40:11.520298 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 16:40:11.520303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 16:40:11.520647 (+ 344us) server_negotiation.cc:407] Connection header received
0619 16:40:11.520740 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 16:40:11.520742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 16:40:11.520799 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 16:40:11.520868 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 16:40:11.521254 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.522067 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.523233 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 16:40:11.523570 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 16:40:11.523662 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 16:40:11.524007 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 16:40:11.524086 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 16:40:11.524167 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 16:40:11.524483 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 16:40:11.524547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 16:40:11.528046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 23017
I20260619 16:40:11.534772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 23092
I20260619 16:40:11.540517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFXRShK/build/debug/bin/kudu with pid 23166
2026-06-19T16:40:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3083 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6183 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284453 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST